2016-10-13 15:40:35,490 INFO [main] hbase.ResourceChecker(147): before: regionserver.TestClusterId#testRewritingClusterIdToPB Thread=4, OpenFileDescriptor=167, MaxFileDescriptor=10240, SystemLoadAverage=318, ProcessCount=284, AvailableMemoryMB=298, ConnectionCount=0 2016-10-13 15:40:35,535 INFO [main] hbase.HBaseTestingUtility(487): Created new mini-cluster data directory: /Users/tyu/1.3-h/hbase-server/target/test-data/7b0c07fa-959d-41eb-9a4b-fdb160a1695f/dfscluster_97fdb37b-c13f-45aa-bebd-e4dcc0a24794, deleteOnExit=true 2016-10-13 15:40:36,731 INFO [main] zookeeper.MiniZooKeeperCluster(276): Started MiniZooKeeperCluster and ran successful 'stat' on client port=51820 2016-10-13 15:40:36,732 INFO [main] hbase.HBaseTestingUtility(737): Setting test.cache.data to /Users/tyu/1.3-h/hbase-server/target/test-data/7b0c07fa-959d-41eb-9a4b-fdb160a1695f/cache_data in system properties and HBase conf 2016-10-13 15:40:36,732 INFO [main] hbase.HBaseTestingUtility(737): Setting hadoop.tmp.dir to /Users/tyu/1.3-h/hbase-server/target/test-data/7b0c07fa-959d-41eb-9a4b-fdb160a1695f/hadoop_tmp in system properties and HBase conf 2016-10-13 15:40:36,732 INFO [main] hbase.HBaseTestingUtility(737): Setting hadoop.log.dir to /Users/tyu/1.3-h/hbase-server/target/test-data/7b0c07fa-959d-41eb-9a4b-fdb160a1695f/hadoop_logs in system properties and HBase conf 2016-10-13 15:40:36,733 INFO [main] hbase.HBaseTestingUtility(737): Setting mapreduce.cluster.local.dir to /Users/tyu/1.3-h/hbase-server/target/test-data/7b0c07fa-959d-41eb-9a4b-fdb160a1695f/mapred_local in system properties and HBase conf 2016-10-13 15:40:36,733 INFO [main] hbase.HBaseTestingUtility(737): Setting mapreduce.cluster.temp.dir to /Users/tyu/1.3-h/hbase-server/target/test-data/7b0c07fa-959d-41eb-9a4b-fdb160a1695f/mapred_temp in system properties and HBase conf 2016-10-13 15:40:36,733 INFO [main] hbase.HBaseTestingUtility(728): read short circuit is OFF 2016-10-13 15:40:36,808 WARN [main] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2016-10-13 15:40:36,915 DEBUG [main] fs.HFileSystem(244): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2016-10-13 15:40:37,523 WARN [main] impl.MetricsConfig(124): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2016-10-13 15:40:37,636 INFO [main] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2016-10-13 15:40:37,671 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2016-10-13 15:40:37,689 INFO [main] log.Slf4jLog(67): Extract jar:file:/Users/tyu/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.5.1/hadoop-hdfs-2.5.1-tests.jar!/webapps/hdfs to /var/folders/4g/2vdss5497xx9blpn2pbqc38r0000gn/T/Jetty_localhost_59173_hdfs____.c74o19/webapp 2016-10-13 15:40:37,800 WARN [main] server.AuthenticationFilter(166): 'signature.secret' configuration not set, using a random value as secret 2016-10-13 15:40:37,824 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59173 2016-10-13 15:40:38,379 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2016-10-13 15:40:38,381 INFO [main] log.Slf4jLog(67): Extract jar:file:/Users/tyu/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.5.1/hadoop-hdfs-2.5.1-tests.jar!/webapps/datanode to /var/folders/4g/2vdss5497xx9blpn2pbqc38r0000gn/T/Jetty_localhost_59176_datanode____.w5agz1/webapp 2016-10-13 15:40:38,454 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59176 2016-10-13 15:40:38,848 INFO [IPC Server handler 4 on 59174] blockmanagement.BlockManager(1753): BLOCK* processReport: from storage DS-159c3408-b48b-4a5d-bfc4-657cae093901 node DatanodeRegistration(127.0.0.1, datanodeUuid=bb9c674b-ccb2-431d-966b-d05351900bac, infoPort=59176, ipcPort=59177, storageInfo=lv=-55;cid=testClusterID;nsid=336749052;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs 2016-10-13 15:40:38,848 INFO [IPC Server handler 4 on 59174] blockmanagement.BlockManager(1753): BLOCK* processReport: from storage DS-0035c63c-edba-4f28-91c9-1a9eaa2e86a4 node DatanodeRegistration(127.0.0.1, datanodeUuid=bb9c674b-ccb2-431d-966b-d05351900bac, infoPort=59176, ipcPort=59177, storageInfo=lv=-55;cid=testClusterID;nsid=336749052;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs 2016-10-13 15:40:38,895 INFO [main] fs.HFileSystem(275): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-13 15:40:38,897 INFO [main] fs.HFileSystem(275): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-13 15:40:39,074 INFO [IPC Server handler 2 on 59174] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59175 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0035c63c-edba-4f28-91c9-1a9eaa2e86a4:NORMAL|RBW]]} size 0 2016-10-13 15:40:39,087 INFO [main] util.FSUtils(747): Created version file at hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca with version=8 2016-10-13 15:40:39,099 INFO [IPC Server handler 8 on 59174] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59175 is added to blk_1073741826_1002{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-159c3408-b48b-4a5d-bfc4-657cae093901:NORMAL|RBW]]} size 0 2016-10-13 15:40:39,110 INFO [IPC Server handler 4 on 59174] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59175 is added to blk_1073741827_1003{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0035c63c-edba-4f28-91c9-1a9eaa2e86a4:NORMAL|RBW]]} size 0 2016-10-13 15:40:39,111 WARN [IPC Server handler 6 on 59174] namenode.FSDirectory(497): DIR* FSDirectory.unprotectedRenameTo: failed to rename /user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/.tmp/hbase.version to /user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/hbase.version because destination exists 2016-10-13 15:40:39,112 DEBUG [main] util.FSUtils(751): Unable to create version file at hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca, retrying java.io.IOException: Unable to move temp version file to hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/hbase.version at org.apache.hadoop.hbase.util.FSUtils.setVersion(FSUtils.java:735) at org.apache.hadoop.hbase.util.FSUtils.setVersion(FSUtils.java:690) at org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1230) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1059) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1037) at org.apache.hadoop.hbase.regionserver.TestClusterId.testRewritingClusterIdToPB(TestClusterId.java:111) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) 2016-10-13 15:40:39,118 INFO [IPC Server handler 7 on 59174] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741825_1001 127.0.0.1:59175 2016-10-13 15:40:39,122 INFO [IPC Server handler 8 on 59174] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741827_1003 127.0.0.1:59175 2016-10-13 15:40:39,131 INFO [IPC Server handler 1 on 59174] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59175 is added to blk_1073741828_1004{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-159c3408-b48b-4a5d-bfc4-657cae093901:NORMAL|RBW]]} size 0 2016-10-13 15:40:39,133 INFO [main] util.FSUtils(747): Created version file at hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca with version=8 2016-10-13 15:40:39,447 INFO [main] client.ConnectionUtils(108): master//10.22.16.34:0 server-side HConnection retries=350 2016-10-13 15:40:39,814 INFO [main] ipc.SimpleRpcScheduler(196): Using fifo as user call queue, count=1 2016-10-13 15:40:39,859 INFO [main] ipc.RpcServer$Listener(653): master//10.22.16.34:0: started 3 reader(s) listening on port=59186 2016-10-13 15:40:40,034 INFO [main] fs.HFileSystem(275): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-13 15:40:40,168 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=master:59186 connecting to ZooKeeper ensemble=localhost:51820 2016-10-13 15:40:40,196 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:591860x0, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:40:40,197 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(663): master:59186-0x157c034513f0000 connected 2016-10-13 15:40:40,338 DEBUG [main] zookeeper.ZKUtil(371): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2016-10-13 15:40:40,338 DEBUG [main] zookeeper.ZKUtil(371): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-10-13 15:40:40,341 INFO [RpcServer.responder] ipc.RpcServer$Responder(976): RpcServer.responder: starting 2016-10-13 15:40:40,341 INFO [RpcServer.listener,port=59186] ipc.RpcServer$Listener(787): RpcServer.listener,port=59186: starting 2016-10-13 15:40:40,342 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=59186 2016-10-13 15:40:40,342 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=1,queue=0,port=59186 2016-10-13 15:40:40,343 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=2,queue=0,port=59186 2016-10-13 15:40:40,343 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=59186 2016-10-13 15:40:40,343 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59186 2016-10-13 15:40:40,343 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=0,queue=0,port=59186 2016-10-13 15:40:40,343 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=1,queue=1,port=59186 2016-10-13 15:40:40,344 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=2,queue=0,port=59186 2016-10-13 15:40:40,344 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=3,queue=1,port=59186 2016-10-13 15:40:40,344 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=4,queue=0,port=59186 2016-10-13 15:40:40,344 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59186 2016-10-13 15:40:40,344 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.replication.handler=0,queue=0,port=59186 2016-10-13 15:40:40,344 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.replication.handler=1,queue=0,port=59186 2016-10-13 15:40:40,345 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.replication.handler=2,queue=0,port=59186 2016-10-13 15:40:40,374 INFO [main] master.HMaster(400): hbase.rootdir=hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca, hbase.cluster.distributed=false 2016-10-13 15:40:40,438 INFO [main] master.HMaster(1797): Adding backup master ZNode /hbase/backup-masters/10.22.16.34,59186,1476398439969 2016-10-13 15:40:40,471 DEBUG [main] zookeeper.ZKUtil(369): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/10.22.16.34,59186,1476398439969 2016-10-13 15:40:40,496 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2016-10-13 15:40:40,496 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(369): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2016-10-13 15:40:40,497 INFO [10.22.16.34:59186.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/10.22.16.34,59186,1476398439969 from backup master directory 2016-10-13 15:40:40,499 DEBUG [main-EventThread] zookeeper.ZKUtil(369): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2016-10-13 15:40:40,500 DEBUG [main-EventThread] master.ActiveMasterManager(126): A master is now available 2016-10-13 15:40:40,500 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/10.22.16.34,59186,1476398439969 2016-10-13 15:40:40,506 WARN [10.22.16.34:59186.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2016-10-13 15:40:40,506 INFO [10.22.16.34:59186.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=10.22.16.34,59186,1476398439969 2016-10-13 15:40:40,514 INFO [main] client.ConnectionUtils(108): regionserver//10.22.16.34:0 server-side HConnection retries=350 2016-10-13 15:40:40,514 INFO [main] ipc.SimpleRpcScheduler(196): Using fifo as user call queue, count=1 2016-10-13 15:40:40,516 INFO [main] ipc.RpcServer$Listener(653): regionserver//10.22.16.34:0: started 3 reader(s) listening on port=59188 2016-10-13 15:40:40,519 INFO [main] fs.HFileSystem(275): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-13 15:40:40,521 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:59188 connecting to ZooKeeper ensemble=localhost:51820 2016-10-13 15:40:40,523 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:591880x0, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:40:40,524 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(663): regionserver:59188-0x157c034513f0001 connected 2016-10-13 15:40:40,524 DEBUG [main] zookeeper.ZKUtil(369): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2016-10-13 15:40:40,525 DEBUG [main] zookeeper.ZKUtil(371): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-10-13 15:40:40,526 INFO [RpcServer.responder] ipc.RpcServer$Responder(976): RpcServer.responder: starting 2016-10-13 15:40:40,526 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=59188 2016-10-13 15:40:40,526 INFO [RpcServer.listener,port=59188] ipc.RpcServer$Listener(787): RpcServer.listener,port=59188: starting 2016-10-13 15:40:40,526 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=1,queue=0,port=59188 2016-10-13 15:40:40,527 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=2,queue=0,port=59188 2016-10-13 15:40:40,528 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=59188 2016-10-13 15:40:40,530 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59188 2016-10-13 15:40:40,530 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=0,queue=0,port=59188 2016-10-13 15:40:40,531 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=1,queue=1,port=59188 2016-10-13 15:40:40,531 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=2,queue=0,port=59188 2016-10-13 15:40:40,531 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=3,queue=1,port=59188 2016-10-13 15:40:40,532 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=4,queue=0,port=59188 2016-10-13 15:40:40,532 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59188 2016-10-13 15:40:40,532 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.replication.handler=0,queue=0,port=59188 2016-10-13 15:40:40,533 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.replication.handler=1,queue=0,port=59188 2016-10-13 15:40:40,533 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.replication.handler=2,queue=0,port=59188 2016-10-13 15:40:40,624 INFO [IPC Server handler 9 on 59174] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59175 is added to blk_1073741829_1005{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0035c63c-edba-4f28-91c9-1a9eaa2e86a4:NORMAL|RBW]]} size 0 2016-10-13 15:40:40,625 DEBUG [10.22.16.34:59186.activeMasterManager] util.FSUtils(899): Created cluster ID file at hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/hbase.id with ID: 30b2093f-8f8a-4ac3-9c48-c66d7e5f9c9d 2016-10-13 15:40:40,626 INFO [IPC Server handler 2 on 59174] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741826_1002 127.0.0.1:59175 2016-10-13 15:40:40,627 DEBUG [10.22.16.34:59186.activeMasterManager] util.FSUtils(862): Rewrote the hbase.id file as pb 2016-10-13 15:40:40,676 INFO [10.22.16.34:59186.activeMasterManager] master.MasterFileSystem(530): BOOTSTRAP: creating hbase:meta region 2016-10-13 15:40:40,680 INFO [10.22.16.34:59186.activeMasterManager] regionserver.HRegion(6473): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', BLOOMFILTER => 'NONE', VERSIONS => '10', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'false', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca Table name == hbase:meta 2016-10-13 15:40:40,717 INFO [M:0;10.22.16.34:59186] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x11f25134 connecting to ZooKeeper ensemble=localhost:51820 2016-10-13 15:40:40,717 INFO [RS:0;10.22.16.34:59188] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0xf77060 connecting to ZooKeeper ensemble=localhost:51820 2016-10-13 15:40:40,721 DEBUG [M:0;10.22.16.34:59186-EventThread] zookeeper.ZooKeeperWatcher(600): hconnection-0x11f251340x0, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:40:40,721 DEBUG [M:0;10.22.16.34:59186-EventThread] zookeeper.ZooKeeperWatcher(663): hconnection-0x11f25134-0x157c034513f0002 connected 2016-10-13 15:40:40,721 DEBUG [RS:0;10.22.16.34:59188-EventThread] zookeeper.ZooKeeperWatcher(600): hconnection-0xf770600x0, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:40:40,721 INFO [M:0;10.22.16.34:59186] client.ZooKeeperRegistry(107): ClusterId read in ZooKeeper is null 2016-10-13 15:40:40,721 DEBUG [RS:0;10.22.16.34:59188-EventThread] zookeeper.ZooKeeperWatcher(663): hconnection-0xf77060-0x157c034513f0003 connected 2016-10-13 15:40:40,722 DEBUG [M:0;10.22.16.34:59186] client.ConnectionManager$HConnectionImplementation(910): clusterid came back null, using default default-cluster 2016-10-13 15:40:40,722 INFO [RS:0;10.22.16.34:59188] client.ZooKeeperRegistry(107): ClusterId read in ZooKeeper is null 2016-10-13 15:40:40,722 DEBUG [RS:0;10.22.16.34:59188] client.ConnectionManager$HConnectionImplementation(910): clusterid came back null, using default default-cluster 2016-10-13 15:40:40,731 INFO [IPC Server handler 9 on 59174] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59175 is added to blk_1073741830_1006{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-159c3408-b48b-4a5d-bfc4-657cae093901:NORMAL|RBW]]} size 0 2016-10-13 15:40:40,770 DEBUG [RS:0;10.22.16.34:59188] ipc.AbstractRpcClient(116): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e08b9ac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-13 15:40:40,770 DEBUG [M:0;10.22.16.34:59186] ipc.AbstractRpcClient(116): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b0a3a44, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-13 15:40:40,773 DEBUG [10.22.16.34:59186.activeMasterManager] regionserver.HRegion(740): Instantiated hbase:meta,,1.1588230740 2016-10-13 15:40:40,970 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(557): Allocating LruBlockCache size=995.60 MB, blockSize=64 KB 2016-10-13 15:40:40,998 DEBUG [StoreOpener-1588230740-1] hfile.CacheConfig(571): Trying to use Internal l2 cache 2016-10-13 15:40:40,999 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(241): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=1071848, freeSize=1042890456, maxSize=1043962304, heapSize=1071848, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-13 15:40:41,098 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(149): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2016-10-13 15:40:41,115 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/meta/1588230740/info 2016-10-13 15:40:41,167 DEBUG [10.22.16.34:59186.activeMasterManager] regionserver.HRegion(4045): Found 0 recovered edits file(s) under hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/meta/1588230740 2016-10-13 15:40:41,194 DEBUG [10.22.16.34:59186.activeMasterManager] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2016-10-13 15:40:41,200 DEBUG [10.22.16.34:59186.activeMasterManager] wal.WALSplitter(743): Wrote region seqId=hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/meta/1588230740/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-13 15:40:41,200 INFO [10.22.16.34:59186.activeMasterManager] regionserver.HRegion(882): Onlined 1588230740; next sequenceid=2 2016-10-13 15:40:41,200 DEBUG [10.22.16.34:59186.activeMasterManager] regionserver.HRegion(1417): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2016-10-13 15:40:41,200 DEBUG [10.22.16.34:59186.activeMasterManager] regionserver.HRegion(1444): Updates disabled for region hbase:meta,,1.1588230740 2016-10-13 15:40:41,202 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(893): Closed info 2016-10-13 15:40:41,202 INFO [10.22.16.34:59186.activeMasterManager] regionserver.HRegion(1548): Closed hbase:meta,,1.1588230740 2016-10-13 15:40:41,317 INFO [IPC Server handler 8 on 59174] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59175 is added to blk_1073741831_1007{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0035c63c-edba-4f28-91c9-1a9eaa2e86a4:NORMAL|RBW]]} size 398 2016-10-13 15:40:41,727 DEBUG [10.22.16.34:59186.activeMasterManager] util.FSTableDescriptors(660): Wrote descriptor into: hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2016-10-13 15:40:41,737 INFO [10.22.16.34:59186.activeMasterManager] fs.HFileSystem(275): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-13 15:40:41,775 INFO [10.22.16.34:59186.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2016-10-13 15:40:41,776 DEBUG [10.22.16.34:59186.activeMasterManager] util.FSTableDescriptors(208): Fetching table descriptors from the filesystem. 2016-10-13 15:40:41,854 INFO [10.22.16.34:59186.activeMasterManager] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x20199f55 connecting to ZooKeeper ensemble=localhost:51820 2016-10-13 15:40:41,858 DEBUG [10.22.16.34:59186.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(600): hconnection-0x20199f550x0, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:40:41,858 DEBUG [10.22.16.34:59186.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(663): hconnection-0x20199f55-0x157c034513f0004 connected 2016-10-13 15:40:41,859 DEBUG [10.22.16.34:59186.activeMasterManager] ipc.AbstractRpcClient(116): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6795025d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-13 15:40:41,926 INFO [10.22.16.34:59186.activeMasterManager] balancer.StochasticLoadBalancer(156): loading config 2016-10-13 15:40:41,970 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(371): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2016-10-13 15:40:41,979 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(371): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2016-10-13 15:40:41,984 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(371): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2016-10-13 15:40:41,985 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(371): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2016-10-13 15:40:42,131 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2016-10-13 15:40:42,131 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2016-10-13 15:40:42,131 INFO [10.22.16.34:59186.activeMasterManager] master.HMaster(624): Server active/primary master=10.22.16.34,59186,1476398439969, sessionid=0x157c034513f0000, setting cluster-up flag (Was=false) 2016-10-13 15:40:42,133 INFO [M:0;10.22.16.34:59186] regionserver.HRegionServer(806): ClusterId : 30b2093f-8f8a-4ac3-9c48-c66d7e5f9c9d 2016-10-13 15:40:42,133 INFO [RS:0;10.22.16.34:59188] regionserver.HRegionServer(806): ClusterId : 30b2093f-8f8a-4ac3-9c48-c66d7e5f9c9d 2016-10-13 15:40:42,197 DEBUG [RS:0;10.22.16.34:59188] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2016-10-13 15:40:42,237 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.RecoverableZooKeeper(584): Node /hbase/flush-table-proc/acquired already exists 2016-10-13 15:40:42,238 INFO [10.22.16.34:59186.activeMasterManager] procedure.ZKProcedureUtil(270): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2016-10-13 15:40:42,239 DEBUG [RS:0;10.22.16.34:59188] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2016-10-13 15:40:42,239 DEBUG [10.22.16.34:59186.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(248): Starting the controller for procedure member:10.22.16.34,59186,1476398439969 2016-10-13 15:40:42,239 DEBUG [RS:0;10.22.16.34:59188] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2016-10-13 15:40:42,244 DEBUG [RS:0;10.22.16.34:59188] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2016-10-13 15:40:42,249 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.RecoverableZooKeeper(584): Node /hbase/online-snapshot/acquired already exists 2016-10-13 15:40:42,250 INFO [10.22.16.34:59186.activeMasterManager] procedure.ZKProcedureUtil(270): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2016-10-13 15:40:42,251 DEBUG [10.22.16.34:59186.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(248): Starting the controller for procedure member:10.22.16.34,59186,1476398439969 2016-10-13 15:40:42,254 INFO [RS:0;10.22.16.34:59188] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2016-10-13 15:40:42,274 INFO [RS:0;10.22.16.34:59188] regionserver.HRegionServer$CompactionChecker(1548): CompactionChecker runs every 1sec 2016-10-13 15:40:42,280 DEBUG [RS:0;10.22.16.34:59188] ipc.AbstractRpcClient(116): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fce4be3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.22.16.34/10.22.16.34:0 2016-10-13 15:40:42,295 DEBUG [RS:0;10.22.16.34:59188] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:RS:0;10.22.16.34:59188 2016-10-13 15:40:42,324 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2016-10-13 15:40:42,324 DEBUG [RS:0;10.22.16.34:59188] zookeeper.ZKUtil(369): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.22.16.34,59188,1476398440518 2016-10-13 15:40:42,325 DEBUG [main-EventThread] zookeeper.ZKUtil(369): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.22.16.34,59188,1476398440518 2016-10-13 15:40:42,326 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.22.16.34,59188,1476398440518 2016-10-13 15:40:42,340 INFO [RS:0;10.22.16.34:59188] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2016-10-13 15:40:42,341 INFO [RS:0;10.22.16.34:59188] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2016-10-13 15:40:42,348 INFO [RS:0;10.22.16.34:59188] regionserver.HRegionServer(2321): reportForDuty to master=10.22.16.34,59186,1476398439969 with port=59188, startcode=1476398440518 2016-10-13 15:40:42,354 INFO [10.22.16.34:59186.activeMasterManager] master.MasterCoprocessorHost(87): System coprocessor loading is enabled 2016-10-13 15:40:42,363 DEBUG [10.22.16.34:59186.activeMasterManager] executor.ExecutorService(101): Starting executor service name=MASTER_OPEN_REGION-10.22.16.34:59186, corePoolSize=5, maxPoolSize=5 2016-10-13 15:40:42,363 DEBUG [10.22.16.34:59186.activeMasterManager] executor.ExecutorService(101): Starting executor service name=MASTER_CLOSE_REGION-10.22.16.34:59186, corePoolSize=5, maxPoolSize=5 2016-10-13 15:40:42,363 DEBUG [10.22.16.34:59186.activeMasterManager] executor.ExecutorService(101): Starting executor service name=MASTER_SERVER_OPERATIONS-10.22.16.34:59186, corePoolSize=5, maxPoolSize=5 2016-10-13 15:40:42,363 DEBUG [10.22.16.34:59186.activeMasterManager] executor.ExecutorService(101): Starting executor service name=MASTER_META_SERVER_OPERATIONS-10.22.16.34:59186, corePoolSize=5, maxPoolSize=5 2016-10-13 15:40:42,363 DEBUG [10.22.16.34:59186.activeMasterManager] executor.ExecutorService(101): Starting executor service name=M_LOG_REPLAY_OPS-10.22.16.34:59186, corePoolSize=10, maxPoolSize=10 2016-10-13 15:40:42,363 DEBUG [10.22.16.34:59186.activeMasterManager] executor.ExecutorService(101): Starting executor service name=MASTER_TABLE_OPERATIONS-10.22.16.34:59186, corePoolSize=1, maxPoolSize=1 2016-10-13 15:40:42,491 INFO [10.22.16.34:59186.activeMasterManager] procedure2.ProcedureExecutor(489): Starting procedure executor threads=9 2016-10-13 15:40:42,492 INFO [10.22.16.34:59186.activeMasterManager] wal.WALProcedureStore(297): Starting WAL Procedure Store lease recovery 2016-10-13 15:40:42,493 WARN [10.22.16.34:59186.activeMasterManager] wal.WALProcedureStore(962): Log directory not found: File hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/MasterProcWALs does not exist. 2016-10-13 15:40:42,534 DEBUG [10.22.16.34:59186.activeMasterManager] wal.WALProcedureStore(840): Roll new state log: 1 2016-10-13 15:40:42,535 INFO [10.22.16.34:59186.activeMasterManager] wal.WALProcedureStore(326): Lease acquired for flushLogId: 1 2016-10-13 15:40:42,536 DEBUG [10.22.16.34:59186.activeMasterManager] wal.WALProcedureStore(343): No state logs to replay. 2016-10-13 15:40:42,536 DEBUG [10.22.16.34:59186.activeMasterManager] procedure2.ProcedureExecutor$1(300): load procedures maxProcId=0 2016-10-13 15:40:42,558 DEBUG [10.22.16.34:59186.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2016-10-13 15:40:42,559 INFO [10.22.16.34:59186.activeMasterManager] zookeeper.RecoverableZooKeeper(120): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:51820 2016-10-13 15:40:42,562 DEBUG [10.22.16.34:59186.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(600): replicationLogCleaner0x0, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:40:42,563 DEBUG [10.22.16.34:59186.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(663): replicationLogCleaner-0x157c034513f0005 connected 2016-10-13 15:40:42,621 DEBUG [RS:0;10.22.16.34:59188] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2016-10-13 15:40:42,624 DEBUG [10.22.16.34:59186.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2016-10-13 15:40:42,626 DEBUG [10.22.16.34:59186.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2016-10-13 15:40:42,641 DEBUG [10.22.16.34:59186.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2016-10-13 15:40:42,641 DEBUG [10.22.16.34:59186.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2016-10-13 15:40:42,642 INFO [10.22.16.34:59186.activeMasterManager] master.ServerManager(1050): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2016-10-13 15:40:42,701 DEBUG [RS:0;10.22.16.34:59188] ipc.RpcClientImpl$Connection(716): Connecting to /10.22.16.34:59186 2016-10-13 15:40:42,734 DEBUG [RpcServer.listener,port=59186] ipc.RpcServer$Listener(898): RpcServer.listener,port=59186: connection from 10.22.16.34:59200; # active connections: 1 2016-10-13 15:40:42,743 INFO [RpcServer.reader=1,bindAddress=10.22.16.34,port=59186] ipc.RpcServer$Connection(1734): Auth successful for tyu.hfs.0 (auth:SIMPLE) 2016-10-13 15:40:42,746 INFO [RpcServer.reader=1,bindAddress=10.22.16.34,port=59186] ipc.RpcServer$Connection(1764): Connection from 10.22.16.34 port: 59200 with version info: version: "1.3.0-SNAPSHOT" url: "git://TYus-MacBook-Pro.local/Users/tyu/1.3-h" revision: "28aaa49c6e39ceba96ebbf26aa400a9852ebdfe6" user: "tyu" date: "Thu Oct 13 15:40:24 PDT 2016" src_checksum: "2ab722632609a1916888f7607a252cd5" version_major: 1 version_minor: 3 2016-10-13 15:40:42,861 INFO [RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59186] master.ServerManager(468): Registering server=10.22.16.34,59188,1476398440518 2016-10-13 15:40:42,911 INFO [10.22.16.34:59186.activeMasterManager] master.ServerManager(1067): Finished waiting for region servers count to settle; checked in 1, slept for 269 ms, expecting minimum of 1, maximum of 1, master is running 2016-10-13 15:40:42,914 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:42,916 INFO [RS:0;10.22.16.34:59188] regionserver.HRegionServer(1383): Config from master: hbase.rootdir=hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca 2016-10-13 15:40:42,916 INFO [RS:0;10.22.16.34:59188] regionserver.HRegionServer(1383): Config from master: fs.defaultFS=hdfs://localhost:59174 2016-10-13 15:40:42,916 INFO [RS:0;10.22.16.34:59188] regionserver.HRegionServer(1383): Config from master: hbase.master.info.port=-1 2016-10-13 15:40:42,917 WARN [RS:0;10.22.16.34:59188] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2016-10-13 15:40:42,917 INFO [RS:0;10.22.16.34:59188] hfile.CacheConfig(266): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1071848, freeSize=1042890456, maxSize=1043962304, heapSize=1071848, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-13 15:40:42,917 DEBUG [RS:0;10.22.16.34:59188] regionserver.HRegionServer(1647): logdir=hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/WALs/10.22.16.34,59188,1476398440518 2016-10-13 15:40:42,945 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:42,968 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:42,991 DEBUG [RS:0;10.22.16.34:59188] regionserver.Replication(156): ReplicationStatisticsThread 300 2016-10-13 15:40:43,057 INFO [RS:0;10.22.16.34:59188] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2016-10-13 15:40:43,082 INFO [RS:0;10.22.16.34:59188] regionserver.MetricsRegionServerWrapperImpl(105): Computing regionserver metrics every 5000 milliseconds 2016-10-13 15:40:43,146 DEBUG [RS:0;10.22.16.34:59188] executor.ExecutorService(101): Starting executor service name=RS_OPEN_REGION-10.22.16.34:59188, corePoolSize=3, maxPoolSize=3 2016-10-13 15:40:43,146 DEBUG [RS:0;10.22.16.34:59188] executor.ExecutorService(101): Starting executor service name=RS_OPEN_META-10.22.16.34:59188, corePoolSize=1, maxPoolSize=1 2016-10-13 15:40:43,146 DEBUG [RS:0;10.22.16.34:59188] executor.ExecutorService(101): Starting executor service name=RS_OPEN_PRIORITY_REGION-10.22.16.34:59188, corePoolSize=3, maxPoolSize=3 2016-10-13 15:40:43,146 DEBUG [RS:0;10.22.16.34:59188] executor.ExecutorService(101): Starting executor service name=RS_CLOSE_REGION-10.22.16.34:59188, corePoolSize=3, maxPoolSize=3 2016-10-13 15:40:43,146 DEBUG [RS:0;10.22.16.34:59188] executor.ExecutorService(101): Starting executor service name=RS_CLOSE_META-10.22.16.34:59188, corePoolSize=1, maxPoolSize=1 2016-10-13 15:40:43,147 DEBUG [RS:0;10.22.16.34:59188] executor.ExecutorService(101): Starting executor service name=RS_LOG_REPLAY_OPS-10.22.16.34:59188, corePoolSize=2, maxPoolSize=2 2016-10-13 15:40:43,147 DEBUG [RS:0;10.22.16.34:59188] executor.ExecutorService(101): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-10.22.16.34:59188, corePoolSize=10, maxPoolSize=10 2016-10-13 15:40:43,147 DEBUG [RS:0;10.22.16.34:59188] executor.ExecutorService(101): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.22.16.34:59188, corePoolSize=3, maxPoolSize=3 2016-10-13 15:40:43,149 DEBUG [RS:0;10.22.16.34:59188] zookeeper.ZKUtil(369): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.22.16.34,59188,1476398440518 2016-10-13 15:40:43,149 INFO [RS:0;10.22.16.34:59188] regionserver.ReplicationSourceManager(248): Current list of replicators: [10.22.16.34,59188,1476398440518] other RSs: [10.22.16.34,59188,1476398440518] 2016-10-13 15:40:43,172 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:43,212 INFO [SplitLogWorker-10.22.16.34:59188] regionserver.SplitLogWorker(134): SplitLogWorker 10.22.16.34,59188,1476398440518 starting 2016-10-13 15:40:43,231 INFO [RS:0;10.22.16.34:59188] regionserver.HeapMemoryManager(191): Starting HeapMemoryTuner chore. 2016-10-13 15:40:43,241 INFO [RS:0;10.22.16.34:59188] regionserver.HRegionServer(1406): Serving as 10.22.16.34,59188,1476398440518, RpcServer on 10.22.16.34/10.22.16.34:59188, sessionid=0x157c034513f0001 2016-10-13 15:40:43,241 DEBUG [RS:0;10.22.16.34:59188] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2016-10-13 15:40:43,242 DEBUG [RS:0;10.22.16.34:59188] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.22.16.34,59188,1476398440518 2016-10-13 15:40:43,242 DEBUG [RS:0;10.22.16.34:59188] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.22.16.34,59188,1476398440518' 2016-10-13 15:40:43,242 DEBUG [RS:0;10.22.16.34:59188] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2016-10-13 15:40:43,242 DEBUG [RS:0;10.22.16.34:59188] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2016-10-13 15:40:43,243 DEBUG [RS:0;10.22.16.34:59188] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2016-10-13 15:40:43,243 DEBUG [RS:0;10.22.16.34:59188] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2016-10-13 15:40:43,243 DEBUG [RS:0;10.22.16.34:59188] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.22.16.34,59188,1476398440518 2016-10-13 15:40:43,243 DEBUG [RS:0;10.22.16.34:59188] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.22.16.34,59188,1476398440518' 2016-10-13 15:40:43,243 DEBUG [RS:0;10.22.16.34:59188] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2016-10-13 15:40:43,243 DEBUG [RS:0;10.22.16.34:59188] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2016-10-13 15:40:43,244 DEBUG [RS:0;10.22.16.34:59188] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2016-10-13 15:40:43,297 INFO [RS:0;10.22.16.34:59188] quotas.RegionServerQuotaManager(53): Quota support disabled 2016-10-13 15:40:43,373 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:43,578 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:43,783 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:43,783 INFO [10.22.16.34:59186.activeMasterManager] zookeeper.MetaTableLocator(532): Deleting hbase:meta region location in ZooKeeper 2016-10-13 15:40:43,787 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.RecoverableZooKeeper(188): Node /hbase/meta-region-server already deleted, retry=false 2016-10-13 15:40:43,810 DEBUG [10.22.16.34:59186.activeMasterManager] master.AssignmentManager(2464): No previous transition plan found (or ignoring an existing plan) for hbase:meta,,1.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=10.22.16.34,59188,1476398440518; 1 (online=1) available servers, forceNewPlan=false 2016-10-13 15:40:43,810 INFO [10.22.16.34:59186.activeMasterManager] master.AssignmentManager(2110): Setting node as OFFLINED in ZooKeeper for region {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2016-10-13 15:40:43,810 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKAssign(205): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Creating (or updating) unassigned node 1588230740 with OFFLINE state 2016-10-13 15:40:43,853 DEBUG [10.22.16.34:59186.activeMasterManager] master.AssignmentManager(2127): Setting table hbase:meta to ENABLED state. 2016-10-13 15:40:43,853 INFO [10.22.16.34:59186.activeMasterManager] zookeeper.ZKTableStateManager(100): Moving table hbase:meta state from null to ENABLED 2016-10-13 15:40:43,877 INFO [10.22.16.34:59186.activeMasterManager] master.AssignmentManager(2142): Assigning hbase:meta,,1.1588230740 to 10.22.16.34,59188,1476398440518 2016-10-13 15:40:43,877 INFO [10.22.16.34:59186.activeMasterManager] master.RegionStates(1118): Transition {1588230740 state=OFFLINE, ts=1476398443810, server=null} to {1588230740 state=PENDING_OPEN, ts=1476398443877, server=10.22.16.34,59188,1476398440518} 2016-10-13 15:40:43,877 DEBUG [10.22.16.34:59186.activeMasterManager] master.ServerManager(976): New admin connection to 10.22.16.34,59188,1476398440518 2016-10-13 15:40:43,976 DEBUG [10.22.16.34:59186.activeMasterManager] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service AdminService, sasl=false 2016-10-13 15:40:43,977 DEBUG [10.22.16.34:59186.activeMasterManager] ipc.RpcClientImpl$Connection(716): Connecting to /10.22.16.34:59188 2016-10-13 15:40:43,978 DEBUG [RpcServer.listener,port=59188] ipc.RpcServer$Listener(898): RpcServer.listener,port=59188: connection from 10.22.16.34:59202; # active connections: 1 2016-10-13 15:40:43,979 INFO [RpcServer.reader=1,bindAddress=10.22.16.34,port=59188] ipc.RpcServer$Connection(1734): Auth successful for tyu (auth:SIMPLE) 2016-10-13 15:40:43,979 INFO [RpcServer.reader=1,bindAddress=10.22.16.34,port=59188] ipc.RpcServer$Connection(1764): Connection from 10.22.16.34 port: 59202 with version info: version: "1.3.0-SNAPSHOT" url: "git://TYus-MacBook-Pro.local/Users/tyu/1.3-h" revision: "28aaa49c6e39ceba96ebbf26aa400a9852ebdfe6" user: "tyu" date: "Thu Oct 13 15:40:24 PDT 2016" src_checksum: "2ab722632609a1916888f7607a252cd5" version_major: 1 version_minor: 3 2016-10-13 15:40:43,984 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59188] regionserver.RSRpcServices(1620): Open hbase:meta,,1.1588230740 2016-10-13 15:40:43,988 DEBUG [RS_OPEN_META-10.22.16.34:59188-0] zookeeper.ZKAssign(805): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Transitioning 1588230740 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2016-10-13 15:40:43,990 DEBUG [RS_OPEN_META-10.22.16.34:59188-0] zookeeper.ZKAssign(880): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Transitioned node 1588230740 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2016-10-13 15:40:43,990 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2016-10-13 15:40:43,990 INFO [RS_OPEN_META-10.22.16.34:59188-0] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2016-10-13 15:40:43,993 DEBUG [10.22.16.34:59186.activeMasterManager] procedure.MasterProcedureScheduler(387): Wake event ProcedureEvent(server crash processing) 2016-10-13 15:40:43,993 DEBUG [AM.ZK.Worker-pool2-t1] master.AssignmentManager(964): Handling RS_ZK_REGION_OPENING, server=10.22.16.34,59188,1476398440518, region=1588230740, current_state={1588230740 state=PENDING_OPEN, ts=1476398443877, server=10.22.16.34,59188,1476398440518} 2016-10-13 15:40:43,993 INFO [10.22.16.34:59186.activeMasterManager] master.ServerManager(698): AssignmentManager hasn't finished failover cleanup; waiting 2016-10-13 15:40:43,993 INFO [AM.ZK.Worker-pool2-t1] master.RegionStates(1118): Transition {1588230740 state=PENDING_OPEN, ts=1476398443877, server=10.22.16.34,59188,1476398440518} to {1588230740 state=OPENING, ts=1476398443993, server=10.22.16.34,59188,1476398440518} 2016-10-13 15:40:43,994 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:43,995 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:43,995 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:43,996 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:43,997 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:43,997 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:43,998 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:43,999 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:43,999 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,000 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,000 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,001 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,001 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,002 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,002 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,002 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,003 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,003 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,004 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,004 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,005 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,005 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,006 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,006 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,007 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,007 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,007 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,008 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,008 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,008 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,009 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,009 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,010 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,010 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,011 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,011 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,012 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,012 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,013 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,014 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,015 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,015 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,016 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,016 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,017 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,017 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,018 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,019 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,019 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,020 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,020 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,021 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,021 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,022 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,022 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,023 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,023 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,024 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,024 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,024 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,025 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,025 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,026 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,026 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,027 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,027 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,028 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,029 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,030 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,030 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,031 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,031 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,032 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,033 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,033 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,034 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,034 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,034 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,035 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,035 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,036 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,036 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,036 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,037 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,037 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,037 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,038 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,038 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,038 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,039 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,039 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,039 INFO [RS_OPEN_META-10.22.16.34:59188-0] wal.FSHLog(527): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.22.16.34%2C59188%2C1476398440518.meta, suffix=.meta, logDir=hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/WALs/10.22.16.34,59188,1476398440518, archiveDir=hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/oldWALs 2016-10-13 15:40:44,039 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,040 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,040 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,041 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,041 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,042 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,042 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,043 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,044 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,044 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,045 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,045 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,045 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,046 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,046 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,047 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,047 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,047 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,048 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,048 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,049 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,049 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,049 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,050 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,050 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,051 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,051 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,052 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,052 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,053 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,053 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,054 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,055 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,055 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,056 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,056 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,056 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,057 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,058 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,058 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,058 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,059 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,059 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,060 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,060 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,060 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,061 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,061 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,062 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,062 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,062 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,063 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,063 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,064 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,064 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,064 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,065 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,065 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,066 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,066 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,067 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,067 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,067 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,068 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,068 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,068 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,069 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,069 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,070 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,070 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,070 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,071 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,071 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,072 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,072 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,072 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,073 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,073 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,073 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,074 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,074 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,074 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,075 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,075 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,075 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,076 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,076 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,076 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,076 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,077 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,077 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,077 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,078 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,078 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,079 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,079 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,080 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,080 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,081 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,081 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,082 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,082 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,082 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,083 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,083 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,084 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,084 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,084 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,085 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,085 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,086 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,086 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,086 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,087 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,087 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,088 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,089 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,089 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,090 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,091 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,091 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,091 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,092 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,092 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,092 INFO [RS_OPEN_META-10.22.16.34:59188-0] wal.FSHLog(1420): Slow sync cost: 7 ms, current pipeline: [] 2016-10-13 15:40:44,092 INFO [RS_OPEN_META-10.22.16.34:59188-0] wal.FSHLog(893): New WAL /user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/WALs/10.22.16.34,59188,1476398440518/10.22.16.34%2C59188%2C1476398440518.meta.1476398444040.meta 2016-10-13 15:40:44,092 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,093 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,093 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,094 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,094 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,094 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,095 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,095 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,096 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,096 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,097 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,097 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,098 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,099 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,099 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,100 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,100 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,101 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,101 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,102 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,102 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,103 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,103 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,103 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,104 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,104 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,105 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,105 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,106 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,106 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,107 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,108 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,108 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,109 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,109 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,109 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,110 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,110 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,110 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,111 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,111 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,111 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,112 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,112 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,112 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,113 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,113 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,113 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,114 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,114 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,115 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,115 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,116 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,116 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,117 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,117 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,118 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,118 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,118 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,119 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,119 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,119 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,120 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,120 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,121 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,121 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,122 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,122 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,122 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,123 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,123 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,123 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,124 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,125 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,125 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,125 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,126 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,126 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,127 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,127 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,128 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,128 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,128 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,129 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,129 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,130 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,130 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,131 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,131 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,132 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,132 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,133 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,133 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,133 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,134 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,134 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,135 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,135 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,135 DEBUG [RS_OPEN_META-10.22.16.34:59188-0] regionserver.HRegion(6662): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2016-10-13 15:40:44,136 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,136 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,137 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,137 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,138 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,139 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,139 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,140 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,140 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,141 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,142 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,142 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,143 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,143 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,144 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,144 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,145 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,145 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,146 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,146 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,147 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,147 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,148 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,149 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,149 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,150 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,150 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,151 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,151 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,152 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,152 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,153 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,153 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,154 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,154 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,155 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,155 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,156 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,156 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,157 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,157 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,158 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,158 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,159 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,159 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,160 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,160 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,161 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,161 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,162 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,162 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,163 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,163 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,164 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,164 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,165 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,165 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,166 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,166 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,167 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,167 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,168 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,168 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,169 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,169 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,170 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,170 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,171 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,171 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,172 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,172 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,173 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,173 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,174 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,174 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,175 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,175 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,176 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,176 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,177 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,177 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,178 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,178 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,179 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,179 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,180 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,180 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,181 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,181 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,182 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,182 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,183 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,183 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,184 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,184 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,185 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,185 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,186 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,186 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,186 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,187 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,187 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,188 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,188 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,189 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,189 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,190 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,190 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,191 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,191 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,191 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,192 DEBUG [RS_OPEN_META-10.22.16.34:59188-0] coprocessor.CoprocessorHost(182): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2016-10-13 15:40:44,192 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,192 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,193 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,193 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,194 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,194 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,195 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,195 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,195 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,196 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,196 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,197 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,197 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,198 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,198 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,199 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,199 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,199 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,200 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,200 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,201 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,201 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,202 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,202 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,203 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,203 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,204 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,204 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,204 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,205 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,205 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,205 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,206 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,206 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,206 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,207 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,207 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,208 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,208 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,209 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,209 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,210 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,210 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,210 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,211 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,211 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,212 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,212 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,212 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,213 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,213 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,214 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,214 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,215 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,215 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,216 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,216 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,216 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,217 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,217 DEBUG [RS_OPEN_META-10.22.16.34:59188-0] regionserver.HRegion(8041): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2016-10-13 15:40:44,217 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,218 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,218 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,219 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,219 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,220 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,220 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,221 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,221 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,222 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,222 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,222 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,223 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,223 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,224 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,224 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,224 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,225 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,225 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,225 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,226 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,226 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,226 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,227 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,227 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,228 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,228 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,228 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,229 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,229 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,230 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,230 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,231 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,231 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,232 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,232 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,233 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,233 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,233 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,234 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,234 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,235 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,235 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,236 INFO [RS_OPEN_META-10.22.16.34:59188-0] regionserver.RegionCoprocessorHost(367): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2016-10-13 15:40:44,236 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,236 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,237 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,237 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,238 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,238 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,239 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,239 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,239 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,240 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,240 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,241 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,241 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,242 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,242 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,243 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,243 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,243 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,244 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,244 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,245 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,245 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,245 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,246 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,246 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,247 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,247 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,247 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,248 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,248 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,248 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,249 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,249 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,250 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,250 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,250 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,251 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,251 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,251 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,252 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,252 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,252 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,253 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,253 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,254 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,254 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,254 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,255 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,255 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,256 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,256 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,257 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,257 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,257 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,258 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,258 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,258 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,258 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,259 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,259 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,259 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,260 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,260 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,260 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,260 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,261 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,261 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,261 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,262 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,262 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,262 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,263 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,263 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,264 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,264 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,265 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,265 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,266 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,266 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,266 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,267 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,267 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,267 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,268 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,268 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,268 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,268 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,269 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,269 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,269 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,269 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,270 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,270 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,271 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,271 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,272 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,272 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,272 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,273 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,273 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,274 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,274 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,275 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,275 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,275 DEBUG [RS_OPEN_META-10.22.16.34:59188-0] regionserver.MetricsRegionSourceImpl(74): Creating new MetricsRegionSourceImpl for table meta 1588230740 2016-10-13 15:40:44,275 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,276 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,276 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,276 DEBUG [RS_OPEN_META-10.22.16.34:59188-0] regionserver.HRegion(740): Instantiated hbase:meta,,1.1588230740 2016-10-13 15:40:44,277 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,277 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,278 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,278 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,278 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,279 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,279 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,280 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,280 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,281 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,281 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,281 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,282 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,282 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,283 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,283 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,284 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,284 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,284 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,285 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,285 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,286 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,286 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,287 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,287 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,288 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,288 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,288 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,289 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,289 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,290 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,290 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,291 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,291 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,292 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,292 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,292 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,293 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,294 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,294 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(241): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=1071848, freeSize=1042890456, maxSize=1043962304, heapSize=1071848, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-13 15:40:44,294 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(149): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2016-10-13 15:40:44,294 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,295 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,295 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/meta/1588230740/info 2016-10-13 15:40:44,296 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,296 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,297 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,297 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,298 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,298 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,299 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,299 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,299 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,300 DEBUG [RS_OPEN_META-10.22.16.34:59188-0] regionserver.HRegion(4045): Found 0 recovered edits file(s) under hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/meta/1588230740 2016-10-13 15:40:44,300 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,300 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,300 INFO [RS:0;10.22.16.34:59188] wal.FSHLog(527): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.22.16.34%2C59188%2C1476398440518, suffix=, logDir=hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/WALs/10.22.16.34,59188,1476398440518, archiveDir=hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/oldWALs 2016-10-13 15:40:44,301 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,301 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,302 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,302 DEBUG [RS_OPEN_META-10.22.16.34:59188-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2016-10-13 15:40:44,302 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,303 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,304 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,304 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,305 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,306 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,308 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,311 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,313 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,313 DEBUG [RS_OPEN_META-10.22.16.34:59188-0] wal.WALSplitter(743): Wrote region seqId=hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/meta/1588230740/recovered.edits/3.seqid to file, newSeqId=3, maxSeqId=2 2016-10-13 15:40:44,313 INFO [RS:0;10.22.16.34:59188] wal.FSHLog(1420): Slow sync cost: 9 ms, current pipeline: [] 2016-10-13 15:40:44,314 INFO [RS:0;10.22.16.34:59188] wal.FSHLog(893): New WAL /user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/WALs/10.22.16.34,59188,1476398440518/10.22.16.34%2C59188%2C1476398440518.1476398444301 2016-10-13 15:40:44,314 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,314 INFO [RS_OPEN_META-10.22.16.34:59188-0] regionserver.HRegion(882): Onlined 1588230740; next sequenceid=3 2016-10-13 15:40:44,314 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,315 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,316 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,316 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,316 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,317 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,318 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,318 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,319 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,319 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,319 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,320 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,320 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,320 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,320 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,321 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,321 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,321 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,322 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,322 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,322 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,323 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,323 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,323 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,323 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,324 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,324 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,324 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,325 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,325 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,325 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,326 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,326 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,326 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,327 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,327 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,327 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,328 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,328 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,329 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,329 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,329 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,330 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,330 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,330 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,331 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,331 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,331 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,332 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,332 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,332 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,332 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,333 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,333 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,333 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,334 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,334 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,334 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,334 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,335 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,335 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,335 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,336 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,336 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,336 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,336 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,337 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,337 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,337 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,337 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,338 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,338 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,338 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,338 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,339 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,339 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,339 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,339 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,340 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,340 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,340 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,340 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,341 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,341 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,341 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,341 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,342 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,342 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,342 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,342 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,343 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,343 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,343 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,343 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,344 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,344 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,344 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,345 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,345 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,345 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,345 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,346 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,346 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,346 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,347 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,347 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,347 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,347 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,348 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,348 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,348 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,348 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,348 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,349 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,349 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,349 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,350 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,350 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,350 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,350 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,351 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,351 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,351 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,351 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,352 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,352 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,352 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,352 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,353 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,353 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,353 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,354 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,354 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,354 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,354 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,355 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,355 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,355 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,355 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,356 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,356 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,356 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,356 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,357 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,357 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,357 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,357 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,358 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,358 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,358 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,359 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,359 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,359 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,359 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,360 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,360 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,360 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,360 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,361 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,361 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,361 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,362 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,362 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,362 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,363 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,363 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,363 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,364 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,364 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,364 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,365 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,365 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,365 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,366 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,366 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,366 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,366 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,367 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,367 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,368 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,368 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,368 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,368 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,369 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,369 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,369 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,370 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,370 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,370 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,371 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,371 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,371 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,371 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,372 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,372 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,372 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,372 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,373 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,373 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,373 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,374 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,374 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,374 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,375 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,375 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,375 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,376 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,376 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,376 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,377 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,377 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,377 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,378 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,378 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,379 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,379 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,379 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,380 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,380 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,381 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,381 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,381 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,382 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,382 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,382 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,382 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,383 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,383 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,399 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,399 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,400 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,400 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,401 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,401 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,402 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,402 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,402 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,403 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,403 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,404 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,404 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,404 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,405 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,405 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,405 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,406 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,406 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,406 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,406 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,407 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,407 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,407 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,407 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,408 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,408 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,408 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,408 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,409 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,409 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,409 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,409 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,410 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,410 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,410 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,410 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,410 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,411 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,411 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,411 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,411 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,412 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,412 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,412 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,412 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,413 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,413 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,413 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,414 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,414 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,414 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,415 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,415 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,415 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,415 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,416 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,416 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,416 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,417 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,417 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,417 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,417 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,418 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,418 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,418 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,418 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,419 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,419 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,419 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,420 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,420 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,420 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,420 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,421 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,421 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,421 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,422 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,422 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,422 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,422 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,422 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,423 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,423 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,423 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,423 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,423 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,424 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,424 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,424 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,424 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,424 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,425 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,425 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,425 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,425 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,425 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,426 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,426 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,426 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,426 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,426 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,427 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,427 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,427 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,427 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,427 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,428 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,428 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,428 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,428 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,429 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,429 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,429 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,429 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,429 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,430 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,430 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,430 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,430 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,430 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,431 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,431 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,431 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,431 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,431 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,432 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,432 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,432 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,432 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,433 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,433 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,433 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,433 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,434 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,434 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,434 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,434 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,434 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,434 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,435 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,435 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,435 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,435 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,435 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,436 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,436 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,436 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,436 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,436 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,437 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,437 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,437 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,437 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,437 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,438 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,438 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,438 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,438 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,438 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,439 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,439 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,439 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,439 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,440 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,440 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,440 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,440 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,440 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,441 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,441 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,441 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,441 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,442 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,442 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,442 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,442 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,442 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,443 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,443 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,443 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,443 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,443 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,444 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,444 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,444 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,444 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,444 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,445 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,445 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,445 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,445 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,446 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,446 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,446 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,446 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,447 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,447 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,447 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,447 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,447 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,448 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,448 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,448 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,448 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,449 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,449 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,449 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,450 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,450 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,450 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,450 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,451 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,451 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,451 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,451 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,452 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,452 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,452 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,452 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,453 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,453 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,454 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,454 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,454 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,455 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,455 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,455 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,455 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,456 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,456 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,456 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,456 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,457 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,457 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,457 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,457 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,458 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,458 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,458 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,458 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,458 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,459 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,459 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,459 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,459 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,460 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,460 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,460 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,461 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,461 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,461 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,461 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,462 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,462 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,462 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,463 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,463 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,463 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,464 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,464 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,465 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,465 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,466 DEBUG [RS_OPEN_META-10.22.16.34:59188-0] zookeeper.ZKAssign(643): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Attempting to retransition opening state of node 1588230740 2016-10-13 15:40:44,466 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,466 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,467 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,467 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,467 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,468 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,468 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1942): Post open deploy tasks for hbase:meta,,1.1588230740 2016-10-13 15:40:44,468 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,469 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,469 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,469 INFO [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(446): Setting hbase:meta region location in ZooKeeper as 10.22.16.34,59188,1476398440518 2016-10-13 15:40:44,469 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,470 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,470 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,470 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,471 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,471 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,472 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,472 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,472 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,472 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,473 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,473 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,474 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,474 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,474 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,475 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,475 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,475 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,476 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,476 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,477 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,477 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,478 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,478 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,478 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,479 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,479 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,480 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,480 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,481 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,481 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,482 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,483 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,483 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,483 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,484 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,485 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,485 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,486 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,486 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,487 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,487 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,488 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,488 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,488 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,489 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,489 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,490 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,490 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,491 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,491 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,491 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,492 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,492 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,493 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,493 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,494 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,494 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,495 DEBUG [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(458): META region location doesn't exist, create it 2016-10-13 15:40:44,495 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(628): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:40:44,496 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1976): Finished post open deploy task for hbase:meta,,1.1588230740 2016-10-13 15:40:44,497 DEBUG [RS_OPEN_META-10.22.16.34:59188-0] zookeeper.ZKAssign(805): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Transitioning 1588230740 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2016-10-13 15:40:44,498 DEBUG [RS_OPEN_META-10.22.16.34:59188-0] zookeeper.ZKAssign(880): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Transitioned node 1588230740 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2016-10-13 15:40:44,498 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2016-10-13 15:40:44,498 DEBUG [RS_OPEN_META-10.22.16.34:59188-0] coordination.ZkOpenRegionCoordination(91): Transitioned 1588230740 to OPENED in zk on 10.22.16.34,59188,1476398440518 2016-10-13 15:40:44,499 DEBUG [RS_OPEN_META-10.22.16.34:59188-0] handler.OpenRegionHandler(170): Opened hbase:meta,,1.1588230740 on 10.22.16.34,59188,1476398440518 2016-10-13 15:40:44,499 DEBUG [AM.ZK.Worker-pool2-t2] master.AssignmentManager(964): Handling RS_ZK_REGION_OPENED, server=10.22.16.34,59188,1476398440518, region=1588230740, current_state={1588230740 state=OPENING, ts=1476398443993, server=10.22.16.34,59188,1476398440518} 2016-10-13 15:40:44,499 INFO [AM.ZK.Worker-pool2-t2] master.RegionStates(1118): Transition {1588230740 state=OPENING, ts=1476398443993, server=10.22.16.34,59188,1476398440518} to {1588230740 state=OPEN, ts=1476398444499, server=10.22.16.34,59188,1476398440518} 2016-10-13 15:40:44,509 INFO [AM.ZK.Worker-pool2-t2] coordination.ZkOpenRegionCoordination(348): Handling OPENED of 1588230740 from 10.22.16.34,59186,1476398439969; deleting unassigned node 2016-10-13 15:40:44,510 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2016-10-13 15:40:44,510 DEBUG [AM.ZK.Worker-pool2-t2] zookeeper.ZKAssign(479): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Deleted unassigned node 1588230740 in expected state RS_ZK_REGION_OPENED 2016-10-13 15:40:44,518 DEBUG [AM.ZK.Worker-pool2-t3] master.AssignmentManager$4(1355): Znode hbase:meta,,1.1588230740 deleted, state: {1588230740 state=OPEN, ts=1476398444499, server=10.22.16.34,59188,1476398440518} 2016-10-13 15:40:44,518 DEBUG [AM.ZK.Worker-pool2-t3] master.RegionStates(466): Onlined 1588230740 on 10.22.16.34,59188,1476398440518 2016-10-13 15:40:44,519 INFO [10.22.16.34:59186.activeMasterManager] master.HMaster(986): hbase:meta with replicaId 0 assigned=1, rit=false, location=10.22.16.34,59188,1476398440518 2016-10-13 15:40:44,849 DEBUG [hconnection-0x11f25134-shared--pool6-t1] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-13 15:40:44,849 DEBUG [hconnection-0x11f25134-shared--pool6-t1] ipc.RpcClientImpl$Connection(716): Connecting to /10.22.16.34:59188 2016-10-13 15:40:44,850 DEBUG [RpcServer.listener,port=59188] ipc.RpcServer$Listener(898): RpcServer.listener,port=59188: connection from 10.22.16.34:59205; # active connections: 2 2016-10-13 15:40:44,851 INFO [RpcServer.reader=2,bindAddress=10.22.16.34,port=59188] ipc.RpcServer$Connection(1734): Auth successful for tyu (auth:SIMPLE) 2016-10-13 15:40:44,852 INFO [RpcServer.reader=2,bindAddress=10.22.16.34,port=59188] ipc.RpcServer$Connection(1764): Connection from 10.22.16.34 port: 59205 with version info: version: "1.3.0-SNAPSHOT" url: "git://TYus-MacBook-Pro.local/Users/tyu/1.3-h" revision: "28aaa49c6e39ceba96ebbf26aa400a9852ebdfe6" user: "tyu" date: "Thu Oct 13 15:40:24 PDT 2016" src_checksum: "2ab722632609a1916888f7607a252cd5" version_major: 1 version_minor: 3 2016-10-13 15:40:44,980 INFO [10.22.16.34:59186.activeMasterManager] hbase.MetaMigrationConvertingToPB(165): hbase:meta doesn't have any entries to update. 2016-10-13 15:40:44,981 INFO [10.22.16.34:59186.activeMasterManager] hbase.MetaMigrationConvertingToPB(131): META already up-to date with PB serialization 2016-10-13 15:40:44,985 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKAssign(497): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Deleting any existing unassigned nodes 2016-10-13 15:40:44,986 INFO [10.22.16.34:59186.activeMasterManager] master.AssignmentManager(649): Clean cluster startup. Assigning user regions 2016-10-13 15:40:44,987 INFO [10.22.16.34:59186.activeMasterManager] master.AssignmentManager(507): Joined the cluster in 6ms, failover=false 2016-10-13 15:40:44,989 DEBUG [10.22.16.34:59186.activeMasterManager] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/meta/1588230740/info 2016-10-13 15:40:45,001 INFO [10.22.16.34:59186.activeMasterManager] master.TableNamespaceManager(85): Namespace table not found. Creating... 2016-10-13 15:40:45,012 INFO [10.22.16.34:59186.activeMasterManager] master.HMaster(1582): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2016-10-13 15:40:45,302 DEBUG [10.22.16.34:59186.activeMasterManager] procedure2.ProcedureExecutor(671): Procedure CreateTableProcedure (table=hbase:namespace) id=1 owner=tyu.hfs.0 state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2016-10-13 15:40:45,372 DEBUG [ProcedureExecutor-0] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/hbase:namespace/write-master:591860000000000 2016-10-13 15:40:45,500 INFO [IPC Server handler 4 on 59174] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59175 is added to blk_1073741835_1011{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0035c63c-edba-4f28-91c9-1a9eaa2e86a4:NORMAL|RBW]]} size 0 2016-10-13 15:40:45,502 DEBUG [ProcedureExecutor-0] util.FSTableDescriptors(660): Wrote descriptor into: hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2016-10-13 15:40:45,504 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6473): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/.tmp Table name == hbase:namespace 2016-10-13 15:40:45,513 INFO [IPC Server handler 4 on 59174] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59175 is added to blk_1073741836_1012{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-159c3408-b48b-4a5d-bfc4-657cae093901:NORMAL|RBW]]} size 42 2016-10-13 15:40:45,919 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(740): Instantiated hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f. 2016-10-13 15:40:45,919 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1417): Closing hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f.: disabling compactions & flushes 2016-10-13 15:40:45,919 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1444): Updates disabled for region hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f. 2016-10-13 15:40:45,919 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1548): Closed hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f. 2016-10-13 15:40:46,477 INFO [ProcedureExecutor-0] hbase.MetaTableAccessor(1199): Added 1 2016-10-13 15:40:46,584 INFO [ProcedureExecutor-0] zookeeper.ZKTableStateManager(100): Moving table hbase:namespace state from null to ENABLING 2016-10-13 15:40:46,587 INFO [ProcedureExecutor-0] master.AssignmentManager(1623): Assigning 1 region(s) to 10.22.16.34,59188,1476398440518 2016-10-13 15:40:46,590 DEBUG [ProcedureExecutor-0] zookeeper.ZKAssign(174): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Async create of unassigned node 2639602db19e5a3bbc51fdedaccf045f with OFFLINE state 2016-10-13 15:40:46,591 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2016-10-13 15:40:46,593 DEBUG [main-EventThread] master.OfflineCallback(69): rs={2639602db19e5a3bbc51fdedaccf045f state=OFFLINE, ts=1476398446587, server=null}, server=10.22.16.34,59188,1476398440518 2016-10-13 15:40:46,593 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={2639602db19e5a3bbc51fdedaccf045f state=OFFLINE, ts=1476398446587, server=null}, server=10.22.16.34,59188,1476398440518 2016-10-13 15:40:46,595 DEBUG [ProcedureExecutor-0] master.AssignmentManager(1676): 10.22.16.34,59188,1476398440518 unassigned znodes=1 of total=1; oldCounter=0 2016-10-13 15:40:46,596 INFO [ProcedureExecutor-0] master.RegionStates(1118): Transition {2639602db19e5a3bbc51fdedaccf045f state=OFFLINE, ts=1476398446590, server=null} to {2639602db19e5a3bbc51fdedaccf045f state=PENDING_OPEN, ts=1476398446596, server=10.22.16.34,59188,1476398440518} 2016-10-13 15:40:46,597 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59188] regionserver.RSRpcServices(1620): Open hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f. 2016-10-13 15:40:46,602 DEBUG [RS_OPEN_PRIORITY_REGION-10.22.16.34:59188-0] zookeeper.ZKAssign(805): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Transitioning 2639602db19e5a3bbc51fdedaccf045f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2016-10-13 15:40:46,602 DEBUG [ProcedureExecutor-0] master.AssignmentManager(1818): Bulk assigning done for 10.22.16.34,59188,1476398440518 2016-10-13 15:40:46,602 INFO [ProcedureExecutor-0] zookeeper.ZKTableStateManager(100): Moving table hbase:namespace state from ENABLING to ENABLED 2016-10-13 15:40:46,603 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/2639602db19e5a3bbc51fdedaccf045f 2016-10-13 15:40:46,603 DEBUG [RS_OPEN_PRIORITY_REGION-10.22.16.34:59188-0] zookeeper.ZKAssign(880): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Transitioned node 2639602db19e5a3bbc51fdedaccf045f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2016-10-13 15:40:46,604 DEBUG [RS_OPEN_PRIORITY_REGION-10.22.16.34:59188-0] regionserver.HRegion(6662): Opening region: {ENCODED => 2639602db19e5a3bbc51fdedaccf045f, NAME => 'hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f.', STARTKEY => '', ENDKEY => ''} 2016-10-13 15:40:46,604 DEBUG [AM.ZK.Worker-pool2-t5] master.AssignmentManager(964): Handling RS_ZK_REGION_OPENING, server=10.22.16.34,59188,1476398440518, region=2639602db19e5a3bbc51fdedaccf045f, current_state={2639602db19e5a3bbc51fdedaccf045f state=PENDING_OPEN, ts=1476398446596, server=10.22.16.34,59188,1476398440518} 2016-10-13 15:40:46,604 INFO [AM.ZK.Worker-pool2-t5] master.RegionStates(1118): Transition {2639602db19e5a3bbc51fdedaccf045f state=PENDING_OPEN, ts=1476398446596, server=10.22.16.34,59188,1476398440518} to {2639602db19e5a3bbc51fdedaccf045f state=OPENING, ts=1476398446604, server=10.22.16.34,59188,1476398440518} 2016-10-13 15:40:46,604 DEBUG [RS_OPEN_PRIORITY_REGION-10.22.16.34:59188-0] regionserver.MetricsRegionSourceImpl(74): Creating new MetricsRegionSourceImpl for table namespace 2639602db19e5a3bbc51fdedaccf045f 2016-10-13 15:40:46,605 DEBUG [RS_OPEN_PRIORITY_REGION-10.22.16.34:59188-0] regionserver.HRegion(740): Instantiated hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f. 2016-10-13 15:40:46,608 INFO [StoreOpener-2639602db19e5a3bbc51fdedaccf045f-1] hfile.CacheConfig(241): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=1071848, freeSize=1042890456, maxSize=1043962304, heapSize=1071848, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-13 15:40:46,608 INFO [StoreOpener-2639602db19e5a3bbc51fdedaccf045f-1] compactions.CompactionConfiguration(149): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2016-10-13 15:40:46,609 DEBUG [StoreOpener-2639602db19e5a3bbc51fdedaccf045f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/namespace/2639602db19e5a3bbc51fdedaccf045f/info 2016-10-13 15:40:46,610 DEBUG [RS_OPEN_PRIORITY_REGION-10.22.16.34:59188-0] regionserver.HRegion(4045): Found 0 recovered edits file(s) under hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/namespace/2639602db19e5a3bbc51fdedaccf045f 2016-10-13 15:40:46,612 DEBUG [RS_OPEN_PRIORITY_REGION-10.22.16.34:59188-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2016-10-13 15:40:46,615 DEBUG [RS_OPEN_PRIORITY_REGION-10.22.16.34:59188-0] wal.WALSplitter(743): Wrote region seqId=hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/namespace/2639602db19e5a3bbc51fdedaccf045f/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-13 15:40:46,615 INFO [RS_OPEN_PRIORITY_REGION-10.22.16.34:59188-0] regionserver.HRegion(882): Onlined 2639602db19e5a3bbc51fdedaccf045f; next sequenceid=2 2016-10-13 15:40:46,617 DEBUG [RS_OPEN_PRIORITY_REGION-10.22.16.34:59188-0] zookeeper.ZKAssign(643): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Attempting to retransition opening state of node 2639602db19e5a3bbc51fdedaccf045f 2016-10-13 15:40:46,617 INFO [PostOpenDeployTasks:2639602db19e5a3bbc51fdedaccf045f] regionserver.HRegionServer(1942): Post open deploy tasks for hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f. 2016-10-13 15:40:46,629 INFO [PostOpenDeployTasks:2639602db19e5a3bbc51fdedaccf045f] hbase.MetaTableAccessor(1394): Updated row hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f. with server=10.22.16.34,59188,1476398440518 2016-10-13 15:40:46,629 DEBUG [PostOpenDeployTasks:2639602db19e5a3bbc51fdedaccf045f] regionserver.HRegionServer(1976): Finished post open deploy task for hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f. 2016-10-13 15:40:46,630 DEBUG [RS_OPEN_PRIORITY_REGION-10.22.16.34:59188-0] zookeeper.ZKAssign(805): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Transitioning 2639602db19e5a3bbc51fdedaccf045f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2016-10-13 15:40:46,631 DEBUG [RS_OPEN_PRIORITY_REGION-10.22.16.34:59188-0] zookeeper.ZKAssign(880): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Transitioned node 2639602db19e5a3bbc51fdedaccf045f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2016-10-13 15:40:46,631 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/2639602db19e5a3bbc51fdedaccf045f 2016-10-13 15:40:46,631 DEBUG [RS_OPEN_PRIORITY_REGION-10.22.16.34:59188-0] coordination.ZkOpenRegionCoordination(91): Transitioned 2639602db19e5a3bbc51fdedaccf045f to OPENED in zk on 10.22.16.34,59188,1476398440518 2016-10-13 15:40:46,631 DEBUG [RS_OPEN_PRIORITY_REGION-10.22.16.34:59188-0] handler.OpenRegionHandler(170): Opened hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f. on 10.22.16.34,59188,1476398440518 2016-10-13 15:40:46,631 DEBUG [AM.ZK.Worker-pool2-t6] master.AssignmentManager(964): Handling RS_ZK_REGION_OPENED, server=10.22.16.34,59188,1476398440518, region=2639602db19e5a3bbc51fdedaccf045f, current_state={2639602db19e5a3bbc51fdedaccf045f state=OPENING, ts=1476398446604, server=10.22.16.34,59188,1476398440518} 2016-10-13 15:40:46,632 INFO [AM.ZK.Worker-pool2-t6] master.RegionStates(1118): Transition {2639602db19e5a3bbc51fdedaccf045f state=OPENING, ts=1476398446604, server=10.22.16.34,59188,1476398440518} to {2639602db19e5a3bbc51fdedaccf045f state=OPEN, ts=1476398446632, server=10.22.16.34,59188,1476398440518} 2016-10-13 15:40:46,632 DEBUG [AM.ZK.Worker-pool2-t6] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 2639602db19e5a3bbc51fdedaccf045f from 10.22.16.34,59186,1476398439969; deleting unassigned node 2016-10-13 15:40:46,632 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/2639602db19e5a3bbc51fdedaccf045f 2016-10-13 15:40:46,632 DEBUG [AM.ZK.Worker-pool2-t6] zookeeper.ZKAssign(479): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Deleted unassigned node 2639602db19e5a3bbc51fdedaccf045f in expected state RS_ZK_REGION_OPENED 2016-10-13 15:40:46,633 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2016-10-13 15:40:46,634 DEBUG [AM.ZK.Worker-pool2-t8] master.AssignmentManager$4(1355): Znode hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f. deleted, state: {2639602db19e5a3bbc51fdedaccf045f state=OPEN, ts=1476398446632, server=10.22.16.34,59188,1476398440518} 2016-10-13 15:40:46,634 DEBUG [AM.ZK.Worker-pool2-t8] master.RegionStates(466): Onlined 2639602db19e5a3bbc51fdedaccf045f on 10.22.16.34,59188,1476398440518 2016-10-13 15:40:46,664 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.ZKUtil(371): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2016-10-13 15:40:46,665 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2016-10-13 15:40:46,752 INFO [10.22.16.34:59186.activeMasterManager] master.HMaster(2561): Client=null/null creating {NAME => 'default'} 2016-10-13 15:40:46,827 DEBUG [10.22.16.34:59186.activeMasterManager] procedure2.ProcedureExecutor(671): Procedure CreateNamespaceProcedure (Namespace=default) id=2 owner=tyu.hfs.0 state=RUNNABLE:CREATE_NAMESPACE_PREPARE added to the store. 2016-10-13 15:40:46,927 DEBUG [ProcedureExecutor-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/hbase:namespace/write-master:591860000000000 2016-10-13 15:40:46,927 DEBUG [ProcedureExecutor-0] procedure2.ProcedureExecutor(872): Procedure completed in 1.8040sec: CreateTableProcedure (table=hbase:namespace) id=1 owner=tyu.hfs.0 state=FINISHED 2016-10-13 15:40:47,267 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2016-10-13 15:40:47,270 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node default with data: \x0A\x07default 2016-10-13 15:40:47,481 DEBUG [ProcedureExecutor-0] procedure2.ProcedureExecutor(872): Procedure completed in 621msec: CreateNamespaceProcedure (Namespace=default) id=2 owner=tyu.hfs.0 state=FINISHED 2016-10-13 15:40:47,589 INFO [10.22.16.34:59186.activeMasterManager] master.HMaster(2561): Client=null/null creating {NAME => 'hbase'} 2016-10-13 15:40:47,697 DEBUG [10.22.16.34:59186.activeMasterManager] procedure2.ProcedureExecutor(671): Procedure CreateNamespaceProcedure (Namespace=hbase) id=3 owner=tyu.hfs.0 state=RUNNABLE:CREATE_NAMESPACE_PREPARE added to the store. 2016-10-13 15:40:48,020 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2016-10-13 15:40:48,022 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node default with data: \x0A\x07default 2016-10-13 15:40:48,022 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node hbase with data: \x0A\x05hbase 2016-10-13 15:40:48,233 DEBUG [ProcedureExecutor-2] procedure2.ProcedureExecutor(872): Procedure completed in 537msec: CreateNamespaceProcedure (Namespace=hbase) id=3 owner=tyu.hfs.0 state=FINISHED 2016-10-13 15:40:48,467 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.RecoverableZooKeeper(584): Node /hbase/namespace/default already exists 2016-10-13 15:40:48,468 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2016-10-13 15:40:48,468 DEBUG [10.22.16.34:59186.activeMasterManager] zookeeper.RecoverableZooKeeper(584): Node /hbase/namespace/hbase already exists 2016-10-13 15:40:48,469 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2016-10-13 15:40:48,469 INFO [10.22.16.34:59186.activeMasterManager] master.HMaster(817): Master has completed initialization 2016-10-13 15:40:48,469 DEBUG [10.22.16.34:59186.activeMasterManager] procedure.MasterProcedureScheduler(387): Wake event ProcedureEvent(master initialized) 2016-10-13 15:40:48,484 INFO [10.22.16.34:59186.activeMasterManager] quotas.MasterQuotaManager(63): Quota support disabled 2016-10-13 15:40:48,485 INFO [10.22.16.34:59186.activeMasterManager] zookeeper.ZooKeeperWatcher(231): not a secure deployment, proceeding 2016-10-13 15:40:48,527 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x3a91d146 connecting to ZooKeeper ensemble=localhost:51820 2016-10-13 15:40:48,531 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): hconnection-0x3a91d1460x0, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:40:48,532 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(663): hconnection-0x3a91d146-0x157c034513f0006 connected 2016-10-13 15:40:48,532 DEBUG [main] ipc.AbstractRpcClient(116): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f952d6c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-13 15:40:48,534 DEBUG [htable-pool10-t1] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-13 15:40:48,534 DEBUG [htable-pool10-t1] ipc.RpcClientImpl$Connection(716): Connecting to /10.22.16.34:59188 2016-10-13 15:40:48,535 DEBUG [RpcServer.listener,port=59188] ipc.RpcServer$Listener(898): RpcServer.listener,port=59188: connection from 10.22.16.34:59211; # active connections: 3 2016-10-13 15:40:48,536 INFO [RpcServer.reader=0,bindAddress=10.22.16.34,port=59188] ipc.RpcServer$Connection(1734): Auth successful for tyu (auth:SIMPLE) 2016-10-13 15:40:48,536 INFO [RpcServer.reader=0,bindAddress=10.22.16.34,port=59188] ipc.RpcServer$Connection(1764): Connection from 10.22.16.34 port: 59211 with version info: version: "1.3.0-SNAPSHOT" url: "git://TYus-MacBook-Pro.local/Users/tyu/1.3-h" revision: "28aaa49c6e39ceba96ebbf26aa400a9852ebdfe6" user: "tyu" date: "Thu Oct 13 15:40:24 PDT 2016" src_checksum: "2ab722632609a1916888f7607a252cd5" version_major: 1 version_minor: 3 2016-10-13 15:40:48,539 INFO [main] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x157c034513f0006 2016-10-13 15:40:48,539 DEBUG [main] ipc.RpcClientImpl(1160): Stopping rpc client 2016-10-13 15:40:48,541 DEBUG [RpcServer.reader=0,bindAddress=10.22.16.34,port=59188] ipc.RpcServer$Listener(930): RpcServer.listener,port=59188: DISCONNECTING client 10.22.16.34:59211 because read count=-1. Number of active connections: 3 2016-10-13 15:40:48,600 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0xaa004a0 connecting to ZooKeeper ensemble=localhost:51820 2016-10-13 15:40:48,603 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): hconnection-0xaa004a00x0, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:40:48,603 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(663): hconnection-0xaa004a0-0x157c034513f0007 connected 2016-10-13 15:40:48,604 DEBUG [main] ipc.AbstractRpcClient(116): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@392a04e7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-13 15:40:48,604 INFO [main] hbase.HBaseTestingUtility(1083): Minicluster is up 2016-10-13 15:40:48,604 INFO [main] hbase.HBaseTestingUtility(1251): The hbase.fs.tmp.dir is set to /user/tyu/hbase-staging 2016-10-13 15:40:48,604 INFO [main] hbase.HBaseTestingUtility(1130): Shutting down minicluster 2016-10-13 15:40:48,604 INFO [main] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x157c034513f0007 2016-10-13 15:40:48,605 DEBUG [main] ipc.RpcClientImpl(1160): Stopping rpc client 2016-10-13 15:40:48,605 DEBUG [main] util.JVMClusterUtil(241): Shutting down HBase Cluster 2016-10-13 15:40:48,605 INFO [main] regionserver.HRegionServer(1908): STOPPED: Cluster shutdown requested 2016-10-13 15:40:48,605 INFO [M:0;10.22.16.34:59186] procedure2.ProcedureExecutor(534): Stopping the procedure executor 2016-10-13 15:40:48,605 INFO [M:0;10.22.16.34:59186] wal.WALProcedureStore(233): Stopping the WAL Procedure Store 2016-10-13 15:40:48,606 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2016-10-13 15:40:48,606 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2016-10-13 15:40:48,606 INFO [main] regionserver.HRegionServer(1908): STOPPED: Shutdown requested 2016-10-13 15:40:48,607 DEBUG [main-EventThread] zookeeper.ZKUtil(371): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-10-13 15:40:48,607 INFO [RS:0;10.22.16.34:59188] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2016-10-13 15:40:48,607 DEBUG [main-EventThread] zookeeper.ZKUtil(371): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-10-13 15:40:48,607 INFO [SplitLogWorker-10.22.16.34:59188] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2016-10-13 15:40:48,607 INFO [RS:0;10.22.16.34:59188] regionserver.HeapMemoryManager(202): Stoping HeapMemoryTuner chore. 2016-10-13 15:40:48,607 INFO [SplitLogWorker-10.22.16.34:59188] regionserver.SplitLogWorker(155): SplitLogWorker 10.22.16.34,59188,1476398440518 exiting 2016-10-13 15:40:48,607 INFO [RS:0;10.22.16.34:59188] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2016-10-13 15:40:48,607 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2016-10-13 15:40:48,607 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2016-10-13 15:40:48,607 INFO [RS:0;10.22.16.34:59188] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2016-10-13 15:40:48,609 INFO [RS:0;10.22.16.34:59188] regionserver.HRegionServer(1056): stopping server 10.22.16.34,59188,1476398440518 2016-10-13 15:40:48,609 DEBUG [RS:0;10.22.16.34:59188] zookeeper.MetaTableLocator(622): Stopping MetaTableLocator 2016-10-13 15:40:48,610 DEBUG [RS_CLOSE_REGION-10.22.16.34:59188-0] handler.CloseRegionHandler(119): Processing close of hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f. 2016-10-13 15:40:48,610 INFO [RS:0;10.22.16.34:59188] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x157c034513f0003 2016-10-13 15:40:48,610 DEBUG [RS_CLOSE_REGION-10.22.16.34:59188-0] regionserver.HRegion(1417): Closing hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f.: disabling compactions & flushes 2016-10-13 15:40:48,610 DEBUG [RS_CLOSE_REGION-10.22.16.34:59188-0] regionserver.HRegion(1444): Updates disabled for region hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f. 2016-10-13 15:40:48,610 INFO [RS_CLOSE_REGION-10.22.16.34:59188-0] regionserver.HRegion(2252): Flushing 1/1 column families, memstore=344 B 2016-10-13 15:40:48,610 DEBUG [RS:0;10.22.16.34:59188] ipc.RpcClientImpl(1160): Stopping rpc client 2016-10-13 15:40:48,611 INFO [RS:0;10.22.16.34:59188] regionserver.CompactSplitThread(403): Waiting for Split Thread to finish... 2016-10-13 15:40:48,611 INFO [RS:0;10.22.16.34:59188] regionserver.CompactSplitThread(403): Waiting for Merge Thread to finish... 2016-10-13 15:40:48,611 INFO [RS:0;10.22.16.34:59188] regionserver.CompactSplitThread(403): Waiting for Large Compaction Thread to finish... 2016-10-13 15:40:48,611 INFO [RS:0;10.22.16.34:59188] regionserver.CompactSplitThread(403): Waiting for Small Compaction Thread to finish... 2016-10-13 15:40:48,611 INFO [RS:0;10.22.16.34:59188] regionserver.HRegionServer(1285): Waiting on 2 regions to close 2016-10-13 15:40:48,611 DEBUG [RS:0;10.22.16.34:59188] regionserver.HRegionServer(1289): {2639602db19e5a3bbc51fdedaccf045f=hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f., 1588230740=hbase:meta,,1.1588230740} 2016-10-13 15:40:48,611 DEBUG [RS_CLOSE_META-10.22.16.34:59188-0] handler.CloseRegionHandler(119): Processing close of hbase:meta,,1.1588230740 2016-10-13 15:40:48,612 DEBUG [RS_CLOSE_META-10.22.16.34:59188-0] regionserver.HRegion(1417): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2016-10-13 15:40:48,612 DEBUG [RS_CLOSE_META-10.22.16.34:59188-0] regionserver.HRegion(1444): Updates disabled for region hbase:meta,,1.1588230740 2016-10-13 15:40:48,612 INFO [RS_CLOSE_META-10.22.16.34:59188-0] regionserver.HRegion(2252): Flushing 1/1 column families, memstore=992 B 2016-10-13 15:40:48,664 INFO [IPC Server handler 2 on 59174] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59175 is added to blk_1073741834_1010{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-159c3408-b48b-4a5d-bfc4-657cae093901:NORMAL|RBW]]} size 477 2016-10-13 15:40:48,665 INFO [M:0;10.22.16.34:59186] regionserver.HRegionServer(1056): stopping server 10.22.16.34,59186,1476398439969 2016-10-13 15:40:48,665 DEBUG [M:0;10.22.16.34:59186] zookeeper.MetaTableLocator(622): Stopping MetaTableLocator 2016-10-13 15:40:48,667 INFO [M:0;10.22.16.34:59186] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x157c034513f0002 2016-10-13 15:40:48,667 DEBUG [M:0;10.22.16.34:59186] ipc.RpcClientImpl(1160): Stopping rpc client 2016-10-13 15:40:48,668 DEBUG [RpcServer.reader=2,bindAddress=10.22.16.34,port=59188] ipc.RpcServer$Listener(930): RpcServer.listener,port=59188: DISCONNECTING client 10.22.16.34:59205 because read count=-1. Number of active connections: 2 2016-10-13 15:40:48,678 INFO [M:0;10.22.16.34:59186] regionserver.HRegionServer(1084): stopping server 10.22.16.34,59186,1476398439969; all regions closed. 2016-10-13 15:40:48,678 INFO [M:0;10.22.16.34:59186] hbase.ChoreService(328): Chore service for: 10.22.16.34,59186,1476398439969 had [[ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-10.22.16.34:59186 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.22.16.34,59186,1476398439969-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.22.16.34,59186,1476398439969-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.22.16.34,59186,1476398439969-DoMetricsChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.22.16.34,59186,1476398439969-RegionNormalizerChore Period: 1800000 Unit: MILLISECONDS]] on shutdown 2016-10-13 15:40:48,678 INFO [M:0;10.22.16.34:59186] master.ServerManager(566): Waiting on regionserver(s) to go down 10.22.16.34,59188,1476398440518 2016-10-13 15:40:48,777 INFO [10.22.16.34,59186,1476398439969_splitLogManager__ChoreService_1] hbase.ScheduledChore(183): Chore: SplitLogManager Timeout Monitor was stopped 2016-10-13 15:40:48,962 INFO [IPC Server handler 3 on 59174] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59175 is added to blk_1073741837_1013{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0035c63c-edba-4f28-91c9-1a9eaa2e86a4:NORMAL|RBW]]} size 0 2016-10-13 15:40:48,963 INFO [IPC Server handler 4 on 59174] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59175 is added to blk_1073741838_1014{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-159c3408-b48b-4a5d-bfc4-657cae093901:NORMAL|RBW]]} size 5258 2016-10-13 15:40:48,963 INFO [RS_CLOSE_REGION-10.22.16.34:59188-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=7, memsize=344, hasBloomFilter=true, into tmp file hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/namespace/2639602db19e5a3bbc51fdedaccf045f/.tmp/3cdcd5df35e54374a6f8c14ee7155685 2016-10-13 15:40:49,005 INFO [RS_OPEN_META-10.22.16.34:59188-0-MetaLogRoller] regionserver.LogRoller(176): LogRoller exiting. 2016-10-13 15:40:49,142 DEBUG [RS_CLOSE_REGION-10.22.16.34:59188-0] regionserver.HRegionFileSystem(390): Committing store file hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/namespace/2639602db19e5a3bbc51fdedaccf045f/.tmp/3cdcd5df35e54374a6f8c14ee7155685 as hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/namespace/2639602db19e5a3bbc51fdedaccf045f/info/3cdcd5df35e54374a6f8c14ee7155685 2016-10-13 15:40:49,146 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2016-10-13 15:40:49,147 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2016-10-13 15:40:49,148 INFO [10.22.16.34,59188,1476398440518_ChoreService_1] hbase.ScheduledChore(183): Chore: 10.22.16.34,59188,1476398440518-MemstoreFlusherChore was stopped 2016-10-13 15:40:49,150 INFO [RS_CLOSE_REGION-10.22.16.34:59188-0] regionserver.HStore(989): Added hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/namespace/2639602db19e5a3bbc51fdedaccf045f/info/3cdcd5df35e54374a6f8c14ee7155685, entries=2, sequenceid=7, filesize=4.8 K 2016-10-13 15:40:49,151 INFO [RS_CLOSE_REGION-10.22.16.34:59188-0] regionserver.HRegion(2552): Finished memstore flush of ~344 B/344, currentsize=0 B/0 for region hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f. in 541ms, sequenceid=7, compaction requested=false 2016-10-13 15:40:49,154 INFO [StoreCloserThread-hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f.-1] regionserver.HStore(893): Closed info 2016-10-13 15:40:49,157 INFO [regionserver//10.22.16.34:0.logRoller] regionserver.LogRoller(176): LogRoller exiting. 2016-10-13 15:40:49,157 INFO [regionserver//10.22.16.34:0.leaseChecker] regionserver.Leases(146): regionserver//10.22.16.34:0.leaseChecker closing leases 2016-10-13 15:40:49,157 INFO [regionserver//10.22.16.34:0.leaseChecker] regionserver.Leases(149): regionserver//10.22.16.34:0.leaseChecker closed leases 2016-10-13 15:40:49,159 DEBUG [RS_CLOSE_REGION-10.22.16.34:59188-0] wal.WALSplitter(743): Wrote region seqId=hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/namespace/2639602db19e5a3bbc51fdedaccf045f/recovered.edits/10.seqid to file, newSeqId=10, maxSeqId=2 2016-10-13 15:40:49,161 INFO [RS_CLOSE_REGION-10.22.16.34:59188-0] regionserver.HRegion(1548): Closed hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f. 2016-10-13 15:40:49,161 DEBUG [RS_CLOSE_REGION-10.22.16.34:59188-0] handler.CloseRegionHandler(164): Closed hbase:namespace,,1476398445009.2639602db19e5a3bbc51fdedaccf045f. 2016-10-13 15:40:49,368 INFO [RS_CLOSE_META-10.22.16.34:59188-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=8, memsize=992, hasBloomFilter=false, into tmp file hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/meta/1588230740/.tmp/fd06148779e04e3d9267f15728eb1e2f 2016-10-13 15:40:49,376 DEBUG [RS_CLOSE_META-10.22.16.34:59188-0] regionserver.HRegionFileSystem(390): Committing store file hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/meta/1588230740/.tmp/fd06148779e04e3d9267f15728eb1e2f as hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/meta/1588230740/info/fd06148779e04e3d9267f15728eb1e2f 2016-10-13 15:40:49,382 INFO [RS_CLOSE_META-10.22.16.34:59188-0] regionserver.HStore(989): Added hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/meta/1588230740/info/fd06148779e04e3d9267f15728eb1e2f, entries=4, sequenceid=8, filesize=5.1 K 2016-10-13 15:40:49,383 INFO [RS_CLOSE_META-10.22.16.34:59188-0] regionserver.HRegion(2552): Finished memstore flush of ~992 B/992, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 771ms, sequenceid=8, compaction requested=false 2016-10-13 15:40:49,384 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(893): Closed info 2016-10-13 15:40:49,388 DEBUG [RS_CLOSE_META-10.22.16.34:59188-0] wal.WALSplitter(743): Wrote region seqId=hdfs://localhost:59174/user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/data/hbase/meta/1588230740/recovered.edits/11.seqid to file, newSeqId=11, maxSeqId=3 2016-10-13 15:40:49,388 DEBUG [RS_CLOSE_META-10.22.16.34:59188-0] coprocessor.CoprocessorHost(272): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2016-10-13 15:40:49,389 INFO [RS_CLOSE_META-10.22.16.34:59188-0] regionserver.HRegion(1548): Closed hbase:meta,,1.1588230740 2016-10-13 15:40:49,389 DEBUG [RS_CLOSE_META-10.22.16.34:59188-0] handler.CloseRegionHandler(164): Closed hbase:meta,,1.1588230740 2016-10-13 15:40:49,426 INFO [RS:0;10.22.16.34:59188] regionserver.HRegionServer(1084): stopping server 10.22.16.34,59188,1476398440518; all regions closed. 2016-10-13 15:40:49,426 DEBUG [RS:0;10.22.16.34:59188] wal.FSHLog(1080): Closing WAL writer in /user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/WALs/10.22.16.34,59188,1476398440518 2016-10-13 15:40:49,432 INFO [IPC Server handler 0 on 59174] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59175 is added to blk_1073741832_1008{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-159c3408-b48b-4a5d-bfc4-657cae093901:NORMAL|RBW]]} size 1312 2016-10-13 15:40:49,711 INFO [M:0;10.22.16.34:59186] master.ServerManager(566): Waiting on regionserver(s) to go down 10.22.16.34,59188,1476398440518 2016-10-13 15:40:49,785 WARN [HBase-Metrics2-1] impl.MetricsConfig(124): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2016-10-13 15:40:49,840 DEBUG [RS:0;10.22.16.34:59188] wal.FSHLog(1038): Moved 1 WAL file(s) to /user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/oldWALs 2016-10-13 15:40:49,841 INFO [RS:0;10.22.16.34:59188] wal.FSHLog(1041): Closed WAL: FSHLog 10.22.16.34%2C59188%2C1476398440518.meta:.meta(num 1476398444040) 2016-10-13 15:40:49,841 DEBUG [RS:0;10.22.16.34:59188] wal.FSHLog(1080): Closing WAL writer in /user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/WALs/10.22.16.34,59188,1476398440518 2016-10-13 15:40:49,846 INFO [IPC Server handler 9 on 59174] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59175 is added to blk_1073741833_1009{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0035c63c-edba-4f28-91c9-1a9eaa2e86a4:NORMAL|RBW]]} size 1383 2016-10-13 15:40:50,253 DEBUG [RS:0;10.22.16.34:59188] wal.FSHLog(1038): Moved 1 WAL file(s) to /user/tyu/test-data/0e6c632f-42e6-4495-9a49-2f09184cddca/oldWALs 2016-10-13 15:40:50,253 INFO [RS:0;10.22.16.34:59188] wal.FSHLog(1041): Closed WAL: FSHLog 10.22.16.34%2C59188%2C1476398440518:(num 1476398444301) 2016-10-13 15:40:50,253 DEBUG [RS:0;10.22.16.34:59188] ipc.RpcClientImpl(1160): Stopping rpc client 2016-10-13 15:40:50,254 DEBUG [RpcServer.reader=1,bindAddress=10.22.16.34,port=59186] ipc.RpcServer$Listener(930): RpcServer.listener,port=59186: DISCONNECTING client 10.22.16.34:59200 because read count=-1. Number of active connections: 1 2016-10-13 15:40:50,264 INFO [RS:0;10.22.16.34:59188] regionserver.Leases(146): RS:0;10.22.16.34:59188 closing leases 2016-10-13 15:40:50,265 INFO [RS:0;10.22.16.34:59188] regionserver.Leases(149): RS:0;10.22.16.34:59188 closed leases 2016-10-13 15:40:50,265 INFO [RS:0;10.22.16.34:59188] hbase.ChoreService(328): Chore service for: 10.22.16.34,59188,1476398440518 had [[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region 10.22.16.34,59188,1476398440518 Period: 120000 Unit: MILLISECONDS]] on shutdown 2016-10-13 15:40:50,270 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.22.16.34,59188,1476398440518 2016-10-13 15:40:50,271 INFO [RS:0;10.22.16.34:59188] ipc.RpcServer(2360): Stopping server on 59188 2016-10-13 15:40:50,271 INFO [RpcServer.listener,port=59188] ipc.RpcServer$Listener(834): RpcServer.listener,port=59188: stopping 2016-10-13 15:40:50,272 INFO [RpcServer.responder] ipc.RpcServer$Responder(1077): RpcServer.responder: stopped 2016-10-13 15:40:50,272 INFO [RpcServer.responder] ipc.RpcServer$Responder(980): RpcServer.responder: stopping 2016-10-13 15:40:50,273 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.22.16.34,59188,1476398440518 2016-10-13 15:40:50,273 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.22.16.34,59188,1476398440518 2016-10-13 15:40:50,273 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59188-0x157c034513f0001, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2016-10-13 15:40:50,273 INFO [main-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.22.16.34,59188,1476398440518] 2016-10-13 15:40:50,274 INFO [RS:0;10.22.16.34:59188] regionserver.HRegionServer(1128): stopping server 10.22.16.34,59188,1476398440518; zookeeper connection closed. 2016-10-13 15:40:50,274 INFO [RS:0;10.22.16.34:59188] regionserver.HRegionServer(1131): RS:0;10.22.16.34:59188 exiting 2016-10-13 15:40:50,274 INFO [main-EventThread] master.ServerManager(621): Cluster shutdown set; 10.22.16.34,59188,1476398440518 expired; onlineServers=0 2016-10-13 15:40:50,274 DEBUG [M:0;10.22.16.34:59186] master.HMaster(1181): Stopping service threads 2016-10-13 15:40:50,275 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6cee72e2] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(193): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6cee72e2 2016-10-13 15:40:50,274 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2016-10-13 15:40:50,275 INFO [main] util.JVMClusterUtil(326): Shutdown of 1 master(s) and 1 regionserver(s) complete 2016-10-13 15:40:50,276 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2016-10-13 15:40:50,276 INFO [M:0;10.22.16.34:59186] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x157c034513f0004 2016-10-13 15:40:50,276 DEBUG [main-EventThread] zookeeper.ZKUtil(371): master:59186-0x157c034513f0000, quorum=localhost:51820, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2016-10-13 15:40:50,276 DEBUG [M:0;10.22.16.34:59186] ipc.RpcClientImpl(1160): Stopping rpc client 2016-10-13 15:40:50,287 INFO [M:0;10.22.16.34:59186] hbase.ChoreService(328): Chore service for: 10.22.16.34,59186,1476398439969_splitLogManager_ had [] on shutdown 2016-10-13 15:40:50,288 INFO [M:0;10.22.16.34:59186] flush.MasterFlushTableProcedureManager(79): stop: server shutting down. 2016-10-13 15:40:50,288 INFO [M:0;10.22.16.34:59186] ipc.RpcServer(2360): Stopping server on 59186 2016-10-13 15:40:50,288 INFO [RpcServer.listener,port=59186] ipc.RpcServer$Listener(834): RpcServer.listener,port=59186: stopping 2016-10-13 15:40:50,289 INFO [RpcServer.responder] ipc.RpcServer$Responder(1077): RpcServer.responder: stopped 2016-10-13 15:40:50,289 INFO [RpcServer.responder] ipc.RpcServer$Responder(980): RpcServer.responder: stopping 2016-10-13 15:40:50,291 DEBUG [M:0;10.22.16.34:59186] zookeeper.RecoverableZooKeeper(188): Node /hbase/rs/10.22.16.34,59186,1476398439969 already deleted, retry=false 2016-10-13 15:40:50,292 INFO [M:0;10.22.16.34:59186] regionserver.HRegionServer(1128): stopping server 10.22.16.34,59186,1476398439969; zookeeper connection closed. 2016-10-13 15:40:50,292 INFO [M:0;10.22.16.34:59186] regionserver.HRegionServer(1131): M:0;10.22.16.34:59186 exiting 2016-10-13 15:40:50,296 INFO [main] zookeeper.MiniZooKeeperCluster(319): Shutdown MiniZK cluster with all ZK servers 2016-10-13 15:40:50,297 WARN [main] datanode.DirectoryScanner(375): DirectoryScanner: shutdown has been called 2016-10-13 15:40:50,301 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2016-10-13 15:40:50,305 WARN [DataNode: [[[DISK]file:/Users/tyu/1.3-h/hbase-server/target/test-data/7b0c07fa-959d-41eb-9a4b-fdb160a1695f/dfscluster_97fdb37b-c13f-45aa-bebd-e4dcc0a24794/dfs/data/data1/, [DISK]file:/Users/tyu/1.3-h/hbase-server/target/test-data/7b0c07fa-959d-41eb-9a4b-fdb160a1695f/dfscluster_97fdb37b-c13f-45aa-bebd-e4dcc0a24794/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:59174] datanode.BPServiceActor(722): BPOfferService for Block pool BP-1822426643-10.22.16.34-1476398437363 (Datanode Uuid bb9c674b-ccb2-431d-966b-d05351900bac) service to localhost/127.0.0.1:59174 interrupted 2016-10-13 15:40:50,305 WARN [DataNode: [[[DISK]file:/Users/tyu/1.3-h/hbase-server/target/test-data/7b0c07fa-959d-41eb-9a4b-fdb160a1695f/dfscluster_97fdb37b-c13f-45aa-bebd-e4dcc0a24794/dfs/data/data1/, [DISK]file:/Users/tyu/1.3-h/hbase-server/target/test-data/7b0c07fa-959d-41eb-9a4b-fdb160a1695f/dfscluster_97fdb37b-c13f-45aa-bebd-e4dcc0a24794/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:59174] datanode.BPServiceActor(845): Ending block pool service for: Block pool BP-1822426643-10.22.16.34-1476398437363 (Datanode Uuid bb9c674b-ccb2-431d-966b-d05351900bac) service to localhost/127.0.0.1:59174 2016-10-13 15:40:50,318 WARN [org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor@3e8f7922] blockmanagement.DecommissionManager$Monitor(78): Monitor interrupted: java.lang.InterruptedException: sleep interrupted 2016-10-13 15:40:50,330 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2016-10-13 15:40:50,396 DEBUG [10.22.16.34:59186.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(600): replicationLogCleaner-0x157c034513f0005, quorum=localhost:51820, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2016-10-13 15:40:50,396 DEBUG [10.22.16.34:59186.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(668): replicationLogCleaner-0x157c034513f0005, quorum=localhost:51820, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2016-10-13 15:40:50,450 INFO [main] hbase.HBaseTestingUtility(1143): Minicluster is down 2016-10-13 15:40:50,498 INFO [main] hbase.ResourceChecker(171): after: regionserver.TestClusterId#testRewritingClusterIdToPB Thread=19 (was 4) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: LeaseRenewer:tyu@localhost:59174 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.22.16.34:59186.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: IPC Client (1711693104) connection to localhost/127.0.0.1:59174 from tyu java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:904) org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) Potentially hanging thread: 10.22.16.34:59186.activeMasterManager-SendThread(localhost:51820) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: LeaseRenewer:tyu.hfs.0@localhost:59174 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@59937acd java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:244) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: IPC Parameter Sending Thread #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: SessionTracker java.lang.Object.wait(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) Potentially hanging thread: IPC Client (1711693104) connection to localhost/127.0.0.1:59174 from tyu.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:904) org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: IPC Client (1711693104) connection to localhost/127.0.0.1:59174 from tyu java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:904) org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=214 (was 167) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=292 (was 318), ProcessCount=284 (was 284), AvailableMemoryMB=106 (was 298), ConnectionCount=0 (was 0) 2016-10-13 15:40:50,556 INFO [main] hbase.ResourceChecker(147): before: regionserver.TestClusterId#testClusterId Thread=19, OpenFileDescriptor=214, MaxFileDescriptor=10240, SystemLoadAverage=309, ProcessCount=284, AvailableMemoryMB=107, ConnectionCount=0 2016-10-13 15:40:50,556 INFO [main] hbase.HBaseTestingUtility(419): System.getProperty("hadoop.log.dir") already set to: /Users/tyu/1.3-h/hbase-server/target/test-data/7b0c07fa-959d-41eb-9a4b-fdb160a1695f/hadoop_logs so I do NOT create it in target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4 2016-10-13 15:40:50,556 WARN [main] hbase.HBaseTestingUtility(423): hadoop.log.dir property value differs in configuration and system: Configuration=/Users/tyu/1.3-h/hbase-server/../logs while System=/Users/tyu/1.3-h/hbase-server/target/test-data/7b0c07fa-959d-41eb-9a4b-fdb160a1695f/hadoop_logs Erasing configuration value by system value. 2016-10-13 15:40:50,556 INFO [main] hbase.HBaseTestingUtility(419): System.getProperty("hadoop.tmp.dir") already set to: /Users/tyu/1.3-h/hbase-server/target/test-data/7b0c07fa-959d-41eb-9a4b-fdb160a1695f/hadoop_tmp so I do NOT create it in target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4 2016-10-13 15:40:50,557 WARN [main] hbase.HBaseTestingUtility(423): hadoop.tmp.dir property value differs in configuration and system: Configuration=/tmp/hadoop-tyu while System=/Users/tyu/1.3-h/hbase-server/target/test-data/7b0c07fa-959d-41eb-9a4b-fdb160a1695f/hadoop_tmp Erasing configuration value by system value. 2016-10-13 15:40:50,557 INFO [main] hbase.HBaseTestingUtility(487): Created new mini-cluster data directory: /Users/tyu/1.3-h/hbase-server/target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4/dfscluster_64f81ea3-ebf2-43ae-9724-b2d9ced5cbba, deleteOnExit=true 2016-10-13 15:40:50,560 INFO [main] zookeeper.MiniZooKeeperCluster(276): Started MiniZooKeeperCluster and ran successful 'stat' on client port=55969 2016-10-13 15:40:50,560 INFO [main] hbase.HBaseTestingUtility(737): Setting test.cache.data to /Users/tyu/1.3-h/hbase-server/target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4/cache_data in system properties and HBase conf 2016-10-13 15:40:50,560 INFO [main] hbase.HBaseTestingUtility(737): Setting hadoop.tmp.dir to /Users/tyu/1.3-h/hbase-server/target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4/hadoop_tmp in system properties and HBase conf 2016-10-13 15:40:50,561 INFO [main] hbase.HBaseTestingUtility(737): Setting hadoop.log.dir to /Users/tyu/1.3-h/hbase-server/target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4/hadoop_logs in system properties and HBase conf 2016-10-13 15:40:50,561 INFO [main] hbase.HBaseTestingUtility(737): Setting mapreduce.cluster.local.dir to /Users/tyu/1.3-h/hbase-server/target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4/mapred_local in system properties and HBase conf 2016-10-13 15:40:50,561 INFO [main] hbase.HBaseTestingUtility(737): Setting mapreduce.cluster.temp.dir to /Users/tyu/1.3-h/hbase-server/target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4/mapred_temp in system properties and HBase conf 2016-10-13 15:40:50,561 INFO [main] hbase.HBaseTestingUtility(728): read short circuit is OFF 2016-10-13 15:40:50,561 DEBUG [main] fs.HFileSystem(244): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2016-10-13 15:40:50,593 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2016-10-13 15:40:50,595 INFO [main] log.Slf4jLog(67): Extract jar:file:/Users/tyu/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.5.1/hadoop-hdfs-2.5.1-tests.jar!/webapps/hdfs to /var/folders/4g/2vdss5497xx9blpn2pbqc38r0000gn/T/Jetty_localhost_59226_hdfs____qpwayk/webapp 2016-10-13 15:40:50,670 WARN [main] server.AuthenticationFilter(166): 'signature.secret' configuration not set, using a random value as secret 2016-10-13 15:40:50,670 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59226 2016-10-13 15:40:50,792 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2016-10-13 15:40:50,795 INFO [main] log.Slf4jLog(67): Extract jar:file:/Users/tyu/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.5.1/hadoop-hdfs-2.5.1-tests.jar!/webapps/datanode to /var/folders/4g/2vdss5497xx9blpn2pbqc38r0000gn/T/Jetty_localhost_59229_datanode____k171h8/webapp 2016-10-13 15:40:50,859 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59229 2016-10-13 15:40:50,998 INFO [IPC Server handler 5 on 59227] blockmanagement.BlockManager(1753): BLOCK* processReport: from storage DS-c3973975-1bcc-4f64-82f9-cc946695f6b0 node DatanodeRegistration(127.0.0.1, datanodeUuid=08e7c6d5-66f5-4a51-9f40-6a810af4a1d6, infoPort=59229, ipcPort=59230, storageInfo=lv=-55;cid=testClusterID;nsid=563811925;c=0), blocks: 0, hasStaleStorages: true, processing time: 0 msecs 2016-10-13 15:40:50,998 INFO [IPC Server handler 5 on 59227] blockmanagement.BlockManager(1753): BLOCK* processReport: from storage DS-db86b596-4d7b-4cf7-9974-02d746fadd02 node DatanodeRegistration(127.0.0.1, datanodeUuid=08e7c6d5-66f5-4a51-9f40-6a810af4a1d6, infoPort=59229, ipcPort=59230, storageInfo=lv=-55;cid=testClusterID;nsid=563811925;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs 2016-10-13 15:40:51,081 INFO [main] client.ConnectionUtils(108): regionserver//10.22.16.34:0 server-side HConnection retries=350 2016-10-13 15:40:51,081 INFO [main] ipc.SimpleRpcScheduler(196): Using fifo as user call queue, count=1 2016-10-13 15:40:51,083 INFO [main] ipc.RpcServer$Listener(653): regionserver//10.22.16.34:0: started 3 reader(s) listening on port=59233 2016-10-13 15:40:51,086 INFO [main] fs.HFileSystem(275): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-13 15:40:51,087 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:59233 connecting to ZooKeeper ensemble=localhost:55969 2016-10-13 15:40:51,093 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:592330x0, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:40:51,093 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(663): regionserver:59233-0x157c034877e0000 connected 2016-10-13 15:40:51,094 DEBUG [main] zookeeper.ZKUtil(371): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2016-10-13 15:40:51,094 DEBUG [main] zookeeper.ZKUtil(371): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-10-13 15:40:51,094 INFO [RpcServer.responder] ipc.RpcServer$Responder(976): RpcServer.responder: starting 2016-10-13 15:40:51,094 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=59233 2016-10-13 15:40:51,094 INFO [RpcServer.listener,port=59233] ipc.RpcServer$Listener(787): RpcServer.listener,port=59233: starting 2016-10-13 15:40:51,094 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=1,queue=0,port=59233 2016-10-13 15:40:51,095 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=2,queue=0,port=59233 2016-10-13 15:40:51,095 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=59233 2016-10-13 15:40:51,095 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59233 2016-10-13 15:40:51,095 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=0,queue=0,port=59233 2016-10-13 15:40:51,095 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=1,queue=1,port=59233 2016-10-13 15:40:51,095 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=2,queue=0,port=59233 2016-10-13 15:40:51,096 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=3,queue=1,port=59233 2016-10-13 15:40:51,096 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=4,queue=0,port=59233 2016-10-13 15:40:51,096 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233 2016-10-13 15:40:51,096 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.replication.handler=0,queue=0,port=59233 2016-10-13 15:40:51,096 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.replication.handler=1,queue=0,port=59233 2016-10-13 15:40:51,096 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.replication.handler=2,queue=0,port=59233 2016-10-13 15:40:51,097 INFO [RS:0;10.22.16.34:59233] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x2a1858e8 connecting to ZooKeeper ensemble=localhost:55969 2016-10-13 15:40:51,099 DEBUG [RS:0;10.22.16.34:59233-EventThread] zookeeper.ZooKeeperWatcher(600): hconnection-0x2a1858e80x0, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:40:51,100 DEBUG [RS:0;10.22.16.34:59233-EventThread] zookeeper.ZooKeeperWatcher(663): hconnection-0x2a1858e8-0x157c034877e0001 connected 2016-10-13 15:40:51,100 INFO [RS:0;10.22.16.34:59233] client.ZooKeeperRegistry(107): ClusterId read in ZooKeeper is null 2016-10-13 15:40:51,100 DEBUG [RS:0;10.22.16.34:59233] client.ConnectionManager$HConnectionImplementation(910): clusterid came back null, using default default-cluster 2016-10-13 15:40:51,100 DEBUG [RS:0;10.22.16.34:59233] ipc.AbstractRpcClient(116): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20e03398, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-13 15:40:54,900 WARN [HBase-Metrics2-1] impl.MetricsConfig(124): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2016-10-13 15:40:54,905 ERROR [HBase-Metrics2-1] lib.MethodMetric$2(118): Error invoking method getBlocksTotal java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111) at org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144) at org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:387) at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79) at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:195) at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172) at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57) at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:221) at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:96) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:245) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:229) at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:290) at com.sun.proxy.$Proxy12.postStart(Unknown Source) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:185) at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable.run(JmxCacheBuster.java:109) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3158) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5652) ... 33 more 2016-10-13 15:41:01,102 INFO [main] fs.HFileSystem(275): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-13 15:41:01,104 INFO [main] fs.HFileSystem(275): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-13 15:41:01,121 INFO [IPC Server handler 5 on 59227] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59228 is added to blk_1073741825_1001{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-db86b596-4d7b-4cf7-9974-02d746fadd02:NORMAL|RBW]]} size 7 2016-10-13 15:41:01,530 INFO [main] util.FSUtils(747): Created version file at hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1 with version=8 2016-10-13 15:41:01,534 INFO [main] client.ConnectionUtils(108): master//10.22.16.34:0 server-side HConnection retries=350 2016-10-13 15:41:01,534 INFO [main] ipc.SimpleRpcScheduler(196): Using fifo as user call queue, count=1 2016-10-13 15:41:01,536 INFO [main] ipc.RpcServer$Listener(653): master//10.22.16.34:0: started 3 reader(s) listening on port=59249 2016-10-13 15:41:01,538 INFO [main] fs.HFileSystem(275): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-13 15:41:01,539 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=master:59249 connecting to ZooKeeper ensemble=localhost:55969 2016-10-13 15:41:01,542 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:592490x0, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:41:01,542 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(663): master:59249-0x157c034877e0002 connected 2016-10-13 15:41:01,547 DEBUG [main] zookeeper.ZKUtil(371): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2016-10-13 15:41:01,547 DEBUG [main] zookeeper.ZKUtil(371): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-10-13 15:41:01,548 INFO [RpcServer.responder] ipc.RpcServer$Responder(976): RpcServer.responder: starting 2016-10-13 15:41:01,548 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=59249 2016-10-13 15:41:01,548 INFO [RpcServer.listener,port=59249] ipc.RpcServer$Listener(787): RpcServer.listener,port=59249: starting 2016-10-13 15:41:01,548 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=1,queue=0,port=59249 2016-10-13 15:41:01,548 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=2,queue=0,port=59249 2016-10-13 15:41:01,548 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=59249 2016-10-13 15:41:01,548 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59249 2016-10-13 15:41:01,549 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=0,queue=0,port=59249 2016-10-13 15:41:01,549 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=1,queue=1,port=59249 2016-10-13 15:41:01,549 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=2,queue=0,port=59249 2016-10-13 15:41:01,549 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=3,queue=1,port=59249 2016-10-13 15:41:01,549 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=4,queue=0,port=59249 2016-10-13 15:41:01,549 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59249 2016-10-13 15:41:01,549 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.replication.handler=0,queue=0,port=59249 2016-10-13 15:41:01,549 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.replication.handler=1,queue=0,port=59249 2016-10-13 15:41:01,550 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.replication.handler=2,queue=0,port=59249 2016-10-13 15:41:01,550 INFO [main] master.HMaster(400): hbase.rootdir=hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1, hbase.cluster.distributed=false 2016-10-13 15:41:01,550 INFO [main] master.HMaster(1797): Adding backup master ZNode /hbase/backup-masters/10.22.16.34,59249,1476398461537 2016-10-13 15:41:01,551 DEBUG [main] zookeeper.ZKUtil(369): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/10.22.16.34,59249,1476398461537 2016-10-13 15:41:01,552 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2016-10-13 15:41:01,552 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2016-10-13 15:41:01,553 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(369): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2016-10-13 15:41:01,553 INFO [10.22.16.34:59249.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/10.22.16.34,59249,1476398461537 from backup master directory 2016-10-13 15:41:01,553 DEBUG [main-EventThread] zookeeper.ZKUtil(369): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2016-10-13 15:41:01,553 DEBUG [main-EventThread] master.ActiveMasterManager(126): A master is now available 2016-10-13 15:41:01,554 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/10.22.16.34,59249,1476398461537 2016-10-13 15:41:01,554 WARN [10.22.16.34:59249.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2016-10-13 15:41:01,554 INFO [10.22.16.34:59249.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=10.22.16.34,59249,1476398461537 2016-10-13 15:41:01,571 INFO [IPC Server handler 9 on 59227] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59228 is added to blk_1073741826_1002{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-c3973975-1bcc-4f64-82f9-cc946695f6b0:NORMAL|RBW]]} size 0 2016-10-13 15:41:01,573 DEBUG [10.22.16.34:59249.activeMasterManager] util.FSUtils(899): Created cluster ID file at hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/hbase.id with ID: e7e33356-6327-4d7e-a9c3-07683ab657c4 2016-10-13 15:41:01,574 INFO [main] client.ConnectionUtils(108): regionserver//10.22.16.34:0 server-side HConnection retries=350 2016-10-13 15:41:01,574 INFO [main] ipc.SimpleRpcScheduler(196): Using fifo as user call queue, count=1 2016-10-13 15:41:01,575 INFO [main] ipc.RpcServer$Listener(653): regionserver//10.22.16.34:0: started 3 reader(s) listening on port=59253 2016-10-13 15:41:01,577 INFO [10.22.16.34:59249.activeMasterManager] master.MasterFileSystem(530): BOOTSTRAP: creating hbase:meta region 2016-10-13 15:41:01,577 INFO [10.22.16.34:59249.activeMasterManager] regionserver.HRegion(6473): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', BLOOMFILTER => 'NONE', VERSIONS => '10', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'false', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1 Table name == hbase:meta 2016-10-13 15:41:01,578 INFO [main] fs.HFileSystem(275): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-13 15:41:01,579 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:59253 connecting to ZooKeeper ensemble=localhost:55969 2016-10-13 15:41:01,581 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:592530x0, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:41:01,581 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(663): regionserver:59253-0x157c034877e0003 connected 2016-10-13 15:41:01,582 DEBUG [main] zookeeper.ZKUtil(369): regionserver:59253-0x157c034877e0003, quorum=localhost:55969, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2016-10-13 15:41:01,582 DEBUG [main] zookeeper.ZKUtil(371): regionserver:59253-0x157c034877e0003, quorum=localhost:55969, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-10-13 15:41:01,582 INFO [RpcServer.responder] ipc.RpcServer$Responder(976): RpcServer.responder: starting 2016-10-13 15:41:01,582 INFO [RpcServer.listener,port=59253] ipc.RpcServer$Listener(787): RpcServer.listener,port=59253: starting 2016-10-13 15:41:01,582 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=59253 2016-10-13 15:41:01,583 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=1,queue=0,port=59253 2016-10-13 15:41:01,583 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=2,queue=0,port=59253 2016-10-13 15:41:01,584 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=59253 2016-10-13 15:41:01,584 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59253 2016-10-13 15:41:01,584 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=0,queue=0,port=59253 2016-10-13 15:41:01,585 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=1,queue=1,port=59253 2016-10-13 15:41:01,585 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=2,queue=0,port=59253 2016-10-13 15:41:01,585 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=3,queue=1,port=59253 2016-10-13 15:41:01,585 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=4,queue=0,port=59253 2016-10-13 15:41:01,586 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59253 2016-10-13 15:41:01,586 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.replication.handler=0,queue=0,port=59253 2016-10-13 15:41:01,586 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.replication.handler=1,queue=0,port=59253 2016-10-13 15:41:01,586 INFO [IPC Server handler 8 on 59227] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59228 is added to blk_1073741827_1003{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-db86b596-4d7b-4cf7-9974-02d746fadd02:NORMAL|RBW]]} size 0 2016-10-13 15:41:01,586 DEBUG [main] ipc.RpcExecutor(130): Started RpcServer.FifoWFPBQ.replication.handler=2,queue=0,port=59253 2016-10-13 15:41:01,588 DEBUG [10.22.16.34:59249.activeMasterManager] regionserver.HRegion(740): Instantiated hbase:meta,,1.1588230740 2016-10-13 15:41:01,588 INFO [RS:0;10.22.16.34:59253] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x173241ac connecting to ZooKeeper ensemble=localhost:55969 2016-10-13 15:41:01,588 INFO [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x3fb7cc2a connecting to ZooKeeper ensemble=localhost:55969 2016-10-13 15:41:01,591 DEBUG [RS:0;10.22.16.34:59253-EventThread] zookeeper.ZooKeeperWatcher(600): hconnection-0x173241ac0x0, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:41:01,591 DEBUG [RS:0;10.22.16.34:59253-EventThread] zookeeper.ZooKeeperWatcher(663): hconnection-0x173241ac-0x157c034877e0004 connected 2016-10-13 15:41:01,591 DEBUG [M:0;10.22.16.34:59249-EventThread] zookeeper.ZooKeeperWatcher(600): hconnection-0x3fb7cc2a0x0, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:41:01,591 INFO [RS:0;10.22.16.34:59253] client.ZooKeeperRegistry(107): ClusterId read in ZooKeeper is null 2016-10-13 15:41:01,591 INFO [M:0;10.22.16.34:59249] client.ZooKeeperRegistry(107): ClusterId read in ZooKeeper is null 2016-10-13 15:41:01,591 DEBUG [M:0;10.22.16.34:59249-EventThread] zookeeper.ZooKeeperWatcher(663): hconnection-0x3fb7cc2a-0x157c034877e0005 connected 2016-10-13 15:41:01,591 DEBUG [M:0;10.22.16.34:59249] client.ConnectionManager$HConnectionImplementation(910): clusterid came back null, using default default-cluster 2016-10-13 15:41:01,591 DEBUG [RS:0;10.22.16.34:59253] client.ConnectionManager$HConnectionImplementation(910): clusterid came back null, using default default-cluster 2016-10-13 15:41:01,592 DEBUG [M:0;10.22.16.34:59249] ipc.AbstractRpcClient(116): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@354fa75a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-13 15:41:01,592 DEBUG [RS:0;10.22.16.34:59253] ipc.AbstractRpcClient(116): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1dca2adc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-13 15:41:01,593 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(241): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=1071848, freeSize=1042890456, maxSize=1043962304, heapSize=1071848, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-13 15:41:01,593 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(149): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2016-10-13 15:41:01,594 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/data/hbase/meta/1588230740/info 2016-10-13 15:41:01,595 DEBUG [10.22.16.34:59249.activeMasterManager] regionserver.HRegion(4045): Found 0 recovered edits file(s) under hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/data/hbase/meta/1588230740 2016-10-13 15:41:01,597 DEBUG [10.22.16.34:59249.activeMasterManager] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2016-10-13 15:41:01,600 DEBUG [10.22.16.34:59249.activeMasterManager] wal.WALSplitter(743): Wrote region seqId=hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/data/hbase/meta/1588230740/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-13 15:41:01,600 INFO [10.22.16.34:59249.activeMasterManager] regionserver.HRegion(882): Onlined 1588230740; next sequenceid=2 2016-10-13 15:41:01,601 DEBUG [10.22.16.34:59249.activeMasterManager] regionserver.HRegion(1417): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2016-10-13 15:41:01,601 DEBUG [10.22.16.34:59249.activeMasterManager] regionserver.HRegion(1444): Updates disabled for region hbase:meta,,1.1588230740 2016-10-13 15:41:01,601 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(893): Closed info 2016-10-13 15:41:01,601 INFO [10.22.16.34:59249.activeMasterManager] regionserver.HRegion(1548): Closed hbase:meta,,1.1588230740 2016-10-13 15:41:01,610 INFO [IPC Server handler 7 on 59227] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59228 is added to blk_1073741828_1004{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-c3973975-1bcc-4f64-82f9-cc946695f6b0:NORMAL|RBW]]} size 398 2016-10-13 15:41:02,014 DEBUG [10.22.16.34:59249.activeMasterManager] util.FSTableDescriptors(660): Wrote descriptor into: hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2016-10-13 15:41:02,019 INFO [10.22.16.34:59249.activeMasterManager] fs.HFileSystem(275): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-13 15:41:02,020 INFO [10.22.16.34:59249.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2016-10-13 15:41:02,020 DEBUG [10.22.16.34:59249.activeMasterManager] util.FSTableDescriptors(208): Fetching table descriptors from the filesystem. 2016-10-13 15:41:02,025 INFO [10.22.16.34:59249.activeMasterManager] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x3e930e36 connecting to ZooKeeper ensemble=localhost:55969 2016-10-13 15:41:02,027 DEBUG [10.22.16.34:59249.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(600): hconnection-0x3e930e360x0, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:41:02,028 DEBUG [10.22.16.34:59249.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(663): hconnection-0x3e930e36-0x157c034877e0006 connected 2016-10-13 15:41:02,028 DEBUG [10.22.16.34:59249.activeMasterManager] ipc.AbstractRpcClient(116): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@338f94ac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-13 15:41:02,029 INFO [10.22.16.34:59249.activeMasterManager] balancer.StochasticLoadBalancer(156): loading config 2016-10-13 15:41:02,029 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(371): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2016-10-13 15:41:02,029 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(371): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2016-10-13 15:41:02,030 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(371): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2016-10-13 15:41:02,030 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(371): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2016-10-13 15:41:02,031 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2016-10-13 15:41:02,031 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2016-10-13 15:41:02,031 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59253-0x157c034877e0003, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2016-10-13 15:41:02,031 INFO [10.22.16.34:59249.activeMasterManager] master.HMaster(624): Server active/primary master=10.22.16.34,59249,1476398461537, sessionid=0x157c034877e0002, setting cluster-up flag (Was=false) 2016-10-13 15:41:02,032 INFO [M:0;10.22.16.34:59249] regionserver.HRegionServer(806): ClusterId : e7e33356-6327-4d7e-a9c3-07683ab657c4 2016-10-13 15:41:02,032 INFO [RS:0;10.22.16.34:59233] regionserver.HRegionServer(806): ClusterId : e7e33356-6327-4d7e-a9c3-07683ab657c4 2016-10-13 15:41:02,032 DEBUG [RS:0;10.22.16.34:59233] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2016-10-13 15:41:02,033 INFO [RS:0;10.22.16.34:59253] regionserver.HRegionServer(806): ClusterId : e7e33356-6327-4d7e-a9c3-07683ab657c4 2016-10-13 15:41:02,033 DEBUG [RS:0;10.22.16.34:59253] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2016-10-13 15:41:02,033 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.RecoverableZooKeeper(584): Node /hbase/flush-table-proc/acquired already exists 2016-10-13 15:41:02,033 DEBUG [RS:0;10.22.16.34:59253] zookeeper.RecoverableZooKeeper(584): Node /hbase/flush-table-proc/acquired already exists 2016-10-13 15:41:02,034 INFO [10.22.16.34:59249.activeMasterManager] procedure.ZKProcedureUtil(270): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2016-10-13 15:41:02,034 DEBUG [RS:0;10.22.16.34:59233] zookeeper.RecoverableZooKeeper(584): Node /hbase/flush-table-proc/abort already exists 2016-10-13 15:41:02,034 DEBUG [RS:0;10.22.16.34:59253] zookeeper.RecoverableZooKeeper(584): Node /hbase/flush-table-proc/abort already exists 2016-10-13 15:41:02,034 DEBUG [RS:0;10.22.16.34:59233] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2016-10-13 15:41:02,034 DEBUG [RS:0;10.22.16.34:59253] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2016-10-13 15:41:02,034 DEBUG [RS:0;10.22.16.34:59233] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2016-10-13 15:41:02,034 DEBUG [RS:0;10.22.16.34:59253] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2016-10-13 15:41:02,035 DEBUG [10.22.16.34:59249.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(248): Starting the controller for procedure member:10.22.16.34,59249,1476398461537 2016-10-13 15:41:02,035 DEBUG [RS:0;10.22.16.34:59253] zookeeper.RecoverableZooKeeper(584): Node /hbase/online-snapshot already exists 2016-10-13 15:41:02,036 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.RecoverableZooKeeper(584): Node /hbase/online-snapshot/acquired already exists 2016-10-13 15:41:02,036 DEBUG [RS:0;10.22.16.34:59253] zookeeper.RecoverableZooKeeper(584): Node /hbase/online-snapshot/acquired already exists 2016-10-13 15:41:02,037 DEBUG [RS:0;10.22.16.34:59233] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2016-10-13 15:41:02,037 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.RecoverableZooKeeper(584): Node /hbase/online-snapshot/abort already exists 2016-10-13 15:41:02,037 DEBUG [RS:0;10.22.16.34:59253] zookeeper.RecoverableZooKeeper(584): Node /hbase/online-snapshot/abort already exists 2016-10-13 15:41:02,037 INFO [RS:0;10.22.16.34:59233] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2016-10-13 15:41:02,037 INFO [10.22.16.34:59249.activeMasterManager] procedure.ZKProcedureUtil(270): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2016-10-13 15:41:02,037 DEBUG [RS:0;10.22.16.34:59253] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2016-10-13 15:41:02,037 INFO [RS:0;10.22.16.34:59233] regionserver.HRegionServer$CompactionChecker(1548): CompactionChecker runs every 1sec 2016-10-13 15:41:02,037 INFO [RS:0;10.22.16.34:59253] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2016-10-13 15:41:02,037 DEBUG [RS:0;10.22.16.34:59233] ipc.AbstractRpcClient(116): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d5572ac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.22.16.34/10.22.16.34:0 2016-10-13 15:41:02,037 INFO [RS:0;10.22.16.34:59253] regionserver.HRegionServer$CompactionChecker(1548): CompactionChecker runs every 1sec 2016-10-13 15:41:02,038 DEBUG [RS:0;10.22.16.34:59233] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:RS:0;10.22.16.34:59233 2016-10-13 15:41:02,038 DEBUG [10.22.16.34:59249.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(248): Starting the controller for procedure member:10.22.16.34,59249,1476398461537 2016-10-13 15:41:02,038 DEBUG [RS:0;10.22.16.34:59253] ipc.AbstractRpcClient(116): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5df2b295, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=10.22.16.34/10.22.16.34:0 2016-10-13 15:41:02,038 DEBUG [RS:0;10.22.16.34:59253] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:RS:0;10.22.16.34:59253 2016-10-13 15:41:02,039 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2016-10-13 15:41:02,039 INFO [10.22.16.34:59249.activeMasterManager] master.MasterCoprocessorHost(87): System coprocessor loading is enabled 2016-10-13 15:41:02,039 DEBUG [RS:0;10.22.16.34:59233] zookeeper.ZKUtil(369): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.22.16.34,59233,1476398451084 2016-10-13 15:41:02,039 DEBUG [10.22.16.34:59249.activeMasterManager] executor.ExecutorService(101): Starting executor service name=MASTER_OPEN_REGION-10.22.16.34:59249, corePoolSize=5, maxPoolSize=5 2016-10-13 15:41:02,039 INFO [RS:0;10.22.16.34:59233] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2016-10-13 15:41:02,039 DEBUG [10.22.16.34:59249.activeMasterManager] executor.ExecutorService(101): Starting executor service name=MASTER_CLOSE_REGION-10.22.16.34:59249, corePoolSize=5, maxPoolSize=5 2016-10-13 15:41:02,039 DEBUG [main-EventThread] zookeeper.ZKUtil(369): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.22.16.34,59233,1476398451084 2016-10-13 15:41:02,039 DEBUG [RS:0;10.22.16.34:59253] zookeeper.ZKUtil(369): regionserver:59253-0x157c034877e0003, quorum=localhost:55969, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.22.16.34,59253,1476398461576 2016-10-13 15:41:02,039 DEBUG [10.22.16.34:59249.activeMasterManager] executor.ExecutorService(101): Starting executor service name=MASTER_SERVER_OPERATIONS-10.22.16.34:59249, corePoolSize=5, maxPoolSize=5 2016-10-13 15:41:02,039 INFO [RS:0;10.22.16.34:59233] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2016-10-13 15:41:02,039 DEBUG [10.22.16.34:59249.activeMasterManager] executor.ExecutorService(101): Starting executor service name=MASTER_META_SERVER_OPERATIONS-10.22.16.34:59249, corePoolSize=5, maxPoolSize=5 2016-10-13 15:41:02,039 DEBUG [main-EventThread] zookeeper.ZKUtil(369): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.22.16.34,59253,1476398461576 2016-10-13 15:41:02,039 INFO [RS:0;10.22.16.34:59253] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2016-10-13 15:41:02,039 DEBUG [10.22.16.34:59249.activeMasterManager] executor.ExecutorService(101): Starting executor service name=M_LOG_REPLAY_OPS-10.22.16.34:59249, corePoolSize=10, maxPoolSize=10 2016-10-13 15:41:02,040 INFO [RS:0;10.22.16.34:59253] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2016-10-13 15:41:02,040 INFO [RS:0;10.22.16.34:59233] regionserver.HRegionServer(2321): reportForDuty to master=10.22.16.34,59249,1476398461537 with port=59233, startcode=1476398451084 2016-10-13 15:41:02,040 DEBUG [10.22.16.34:59249.activeMasterManager] executor.ExecutorService(101): Starting executor service name=MASTER_TABLE_OPERATIONS-10.22.16.34:59249, corePoolSize=1, maxPoolSize=1 2016-10-13 15:41:02,040 DEBUG [RS:0;10.22.16.34:59233] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2016-10-13 15:41:02,040 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.22.16.34,59233,1476398451084 2016-10-13 15:41:02,040 INFO [RS:0;10.22.16.34:59253] regionserver.HRegionServer(2321): reportForDuty to master=10.22.16.34,59249,1476398461537 with port=59253, startcode=1476398461576 2016-10-13 15:41:02,040 INFO [10.22.16.34:59249.activeMasterManager] procedure2.ProcedureExecutor(489): Starting procedure executor threads=9 2016-10-13 15:41:02,040 DEBUG [RS:0;10.22.16.34:59233] ipc.RpcClientImpl$Connection(716): Connecting to /10.22.16.34:59249 2016-10-13 15:41:02,040 INFO [10.22.16.34:59249.activeMasterManager] wal.WALProcedureStore(297): Starting WAL Procedure Store lease recovery 2016-10-13 15:41:02,040 DEBUG [RS:0;10.22.16.34:59253] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2016-10-13 15:41:02,040 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/10.22.16.34,59253,1476398461576 2016-10-13 15:41:02,040 DEBUG [RS:0;10.22.16.34:59253] ipc.RpcClientImpl$Connection(716): Connecting to /10.22.16.34:59249 2016-10-13 15:41:02,041 DEBUG [RpcServer.listener,port=59249] ipc.RpcServer$Listener(898): RpcServer.listener,port=59249: connection from 10.22.16.34:59260; # active connections: 1 2016-10-13 15:41:02,042 DEBUG [RpcServer.listener,port=59249] ipc.RpcServer$Listener(898): RpcServer.listener,port=59249: connection from 10.22.16.34:59261; # active connections: 2 2016-10-13 15:41:02,042 WARN [10.22.16.34:59249.activeMasterManager] wal.WALProcedureStore(962): Log directory not found: File hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/MasterProcWALs does not exist. 2016-10-13 15:41:02,042 INFO [RpcServer.reader=1,bindAddress=10.22.16.34,port=59249] ipc.RpcServer$Connection(1734): Auth successful for tyu (auth:SIMPLE) 2016-10-13 15:41:02,042 INFO [RpcServer.reader=2,bindAddress=10.22.16.34,port=59249] ipc.RpcServer$Connection(1734): Auth successful for tyu.hfs.1 (auth:SIMPLE) 2016-10-13 15:41:02,043 INFO [RpcServer.reader=1,bindAddress=10.22.16.34,port=59249] ipc.RpcServer$Connection(1764): Connection from 10.22.16.34 port: 59260 with version info: version: "1.3.0-SNAPSHOT" url: "git://TYus-MacBook-Pro.local/Users/tyu/1.3-h" revision: "28aaa49c6e39ceba96ebbf26aa400a9852ebdfe6" user: "tyu" date: "Thu Oct 13 15:40:24 PDT 2016" src_checksum: "2ab722632609a1916888f7607a252cd5" version_major: 1 version_minor: 3 2016-10-13 15:41:02,043 INFO [RpcServer.reader=2,bindAddress=10.22.16.34,port=59249] ipc.RpcServer$Connection(1764): Connection from 10.22.16.34 port: 59261 with version info: version: "1.3.0-SNAPSHOT" url: "git://TYus-MacBook-Pro.local/Users/tyu/1.3-h" revision: "28aaa49c6e39ceba96ebbf26aa400a9852ebdfe6" user: "tyu" date: "Thu Oct 13 15:40:24 PDT 2016" src_checksum: "2ab722632609a1916888f7607a252cd5" version_major: 1 version_minor: 3 2016-10-13 15:41:02,044 DEBUG [RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59249] ipc.CallRunner(126): RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59249: callId: 0 service: RegionServerStatusService methodName: RegionServerStartup size: 48 connection: 10.22.16.34:59260 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2359) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:358) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:02,044 DEBUG [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=59249] ipc.CallRunner(126): RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=59249: callId: 0 service: RegionServerStatusService methodName: RegionServerStartup size: 48 connection: 10.22.16.34:59261 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2359) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:358) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:02,044 DEBUG [10.22.16.34:59249.activeMasterManager] wal.WALProcedureStore(840): Roll new state log: 1 2016-10-13 15:41:02,045 INFO [10.22.16.34:59249.activeMasterManager] wal.WALProcedureStore(326): Lease acquired for flushLogId: 1 2016-10-13 15:41:02,045 DEBUG [10.22.16.34:59249.activeMasterManager] wal.WALProcedureStore(343): No state logs to replay. 2016-10-13 15:41:02,046 DEBUG [10.22.16.34:59249.activeMasterManager] procedure2.ProcedureExecutor$1(300): load procedures maxProcId=0 2016-10-13 15:41:02,046 DEBUG [10.22.16.34:59249.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2016-10-13 15:41:02,046 INFO [10.22.16.34:59249.activeMasterManager] zookeeper.RecoverableZooKeeper(120): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:55969 2016-10-13 15:41:02,048 DEBUG [10.22.16.34:59249.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(600): replicationLogCleaner0x0, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-13 15:41:02,049 DEBUG [10.22.16.34:59249.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(663): replicationLogCleaner-0x157c034877e0007 connected 2016-10-13 15:41:02,050 DEBUG [10.22.16.34:59249.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2016-10-13 15:41:02,050 DEBUG [10.22.16.34:59249.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2016-10-13 15:41:02,050 DEBUG [10.22.16.34:59249.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2016-10-13 15:41:02,050 DEBUG [10.22.16.34:59249.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2016-10-13 15:41:02,050 INFO [10.22.16.34:59249.activeMasterManager] master.ServerManager(1050): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2016-10-13 15:41:02,052 DEBUG [RS:0;10.22.16.34:59253] regionserver.HRegionServer(2340): Master is not running yet 2016-10-13 15:41:02,052 DEBUG [RS:0;10.22.16.34:59233] regionserver.HRegionServer(2340): Master is not running yet 2016-10-13 15:41:02,052 WARN [RS:0;10.22.16.34:59233] regionserver.HRegionServer(934): reportForDuty failed; sleeping and then retrying. 2016-10-13 15:41:02,052 WARN [RS:0;10.22.16.34:59253] regionserver.HRegionServer(934): reportForDuty failed; sleeping and then retrying. 2016-10-13 15:41:03,053 INFO [RS:0;10.22.16.34:59253] regionserver.HRegionServer(2321): reportForDuty to master=10.22.16.34,59249,1476398461537 with port=59253, startcode=1476398461576 2016-10-13 15:41:03,053 INFO [RS:0;10.22.16.34:59233] regionserver.HRegionServer(2321): reportForDuty to master=10.22.16.34,59249,1476398461537 with port=59233, startcode=1476398451084 2016-10-13 15:41:03,056 INFO [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=59249] master.ServerManager(468): Registering server=10.22.16.34,59253,1476398461576 2016-10-13 15:41:03,056 INFO [RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59249] master.ServerManager(468): Registering server=10.22.16.34,59233,1476398451084 2016-10-13 15:41:03,057 INFO [RS:0;10.22.16.34:59253] regionserver.HRegionServer(1383): Config from master: hbase.rootdir=hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1 2016-10-13 15:41:03,057 INFO [RS:0;10.22.16.34:59233] regionserver.HRegionServer(1383): Config from master: hbase.rootdir=hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1 2016-10-13 15:41:03,057 INFO [RS:0;10.22.16.34:59253] regionserver.HRegionServer(1383): Config from master: fs.defaultFS=hdfs://localhost:59227 2016-10-13 15:41:03,057 INFO [RS:0;10.22.16.34:59253] regionserver.HRegionServer(1383): Config from master: hbase.master.info.port=-1 2016-10-13 15:41:03,057 INFO [RS:0;10.22.16.34:59233] regionserver.HRegionServer(1383): Config from master: fs.defaultFS=hdfs://localhost:59227 2016-10-13 15:41:03,058 WARN [RS:0;10.22.16.34:59253] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2016-10-13 15:41:03,058 INFO [RS:0;10.22.16.34:59233] regionserver.HRegionServer(1383): Config from master: hbase.master.info.port=-1 2016-10-13 15:41:03,058 WARN [RS:0;10.22.16.34:59233] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2016-10-13 15:41:03,058 INFO [RS:0;10.22.16.34:59253] hfile.CacheConfig(266): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1071848, freeSize=1042890456, maxSize=1043962304, heapSize=1071848, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-13 15:41:03,058 INFO [RS:0;10.22.16.34:59233] hfile.CacheConfig(266): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1071848, freeSize=1042890456, maxSize=1043962304, heapSize=1071848, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-13 15:41:03,058 DEBUG [RS:0;10.22.16.34:59253] regionserver.HRegionServer(1647): logdir=hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/WALs/10.22.16.34,59253,1476398461576 2016-10-13 15:41:03,058 DEBUG [RS:0;10.22.16.34:59233] regionserver.HRegionServer(1647): logdir=hdfs://localhost:59227/var/folders/4g/2vdss5497xx9blpn2pbqc38r0000gn/T/hbase-tyu/hbase/WALs/10.22.16.34,59233,1476398451084 2016-10-13 15:41:03,064 DEBUG [RS:0;10.22.16.34:59253] zookeeper.RecoverableZooKeeper(584): Node /hbase/replication/peers already exists 2016-10-13 15:41:03,064 DEBUG [RS:0;10.22.16.34:59233] regionserver.Replication(156): ReplicationStatisticsThread 300 2016-10-13 15:41:03,064 INFO [RS:0;10.22.16.34:59233] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2016-10-13 15:41:03,065 DEBUG [RS:0;10.22.16.34:59253] regionserver.Replication(156): ReplicationStatisticsThread 300 2016-10-13 15:41:03,065 INFO [RS:0;10.22.16.34:59233] regionserver.MetricsRegionServerWrapperImpl(105): Computing regionserver metrics every 5000 milliseconds 2016-10-13 15:41:03,065 INFO [RS:0;10.22.16.34:59253] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2016-10-13 15:41:03,065 INFO [RS:0;10.22.16.34:59253] regionserver.MetricsRegionServerWrapperImpl(105): Computing regionserver metrics every 5000 milliseconds 2016-10-13 15:41:03,080 DEBUG [RS:0;10.22.16.34:59253] executor.ExecutorService(101): Starting executor service name=RS_OPEN_REGION-10.22.16.34:59253, corePoolSize=3, maxPoolSize=3 2016-10-13 15:41:03,080 DEBUG [RS:0;10.22.16.34:59233] executor.ExecutorService(101): Starting executor service name=RS_OPEN_REGION-10.22.16.34:59233, corePoolSize=3, maxPoolSize=3 2016-10-13 15:41:03,080 DEBUG [RS:0;10.22.16.34:59253] executor.ExecutorService(101): Starting executor service name=RS_OPEN_META-10.22.16.34:59253, corePoolSize=1, maxPoolSize=1 2016-10-13 15:41:03,080 DEBUG [RS:0;10.22.16.34:59233] executor.ExecutorService(101): Starting executor service name=RS_OPEN_META-10.22.16.34:59233, corePoolSize=1, maxPoolSize=1 2016-10-13 15:41:03,080 DEBUG [RS:0;10.22.16.34:59253] executor.ExecutorService(101): Starting executor service name=RS_OPEN_PRIORITY_REGION-10.22.16.34:59253, corePoolSize=3, maxPoolSize=3 2016-10-13 15:41:03,080 DEBUG [RS:0;10.22.16.34:59233] executor.ExecutorService(101): Starting executor service name=RS_OPEN_PRIORITY_REGION-10.22.16.34:59233, corePoolSize=3, maxPoolSize=3 2016-10-13 15:41:03,080 DEBUG [RS:0;10.22.16.34:59253] executor.ExecutorService(101): Starting executor service name=RS_CLOSE_REGION-10.22.16.34:59253, corePoolSize=3, maxPoolSize=3 2016-10-13 15:41:03,080 DEBUG [RS:0;10.22.16.34:59233] executor.ExecutorService(101): Starting executor service name=RS_CLOSE_REGION-10.22.16.34:59233, corePoolSize=3, maxPoolSize=3 2016-10-13 15:41:03,080 DEBUG [RS:0;10.22.16.34:59253] executor.ExecutorService(101): Starting executor service name=RS_CLOSE_META-10.22.16.34:59253, corePoolSize=1, maxPoolSize=1 2016-10-13 15:41:03,080 DEBUG [RS:0;10.22.16.34:59233] executor.ExecutorService(101): Starting executor service name=RS_CLOSE_META-10.22.16.34:59233, corePoolSize=1, maxPoolSize=1 2016-10-13 15:41:03,081 DEBUG [RS:0;10.22.16.34:59253] executor.ExecutorService(101): Starting executor service name=RS_LOG_REPLAY_OPS-10.22.16.34:59253, corePoolSize=2, maxPoolSize=2 2016-10-13 15:41:03,081 DEBUG [RS:0;10.22.16.34:59233] executor.ExecutorService(101): Starting executor service name=RS_LOG_REPLAY_OPS-10.22.16.34:59233, corePoolSize=2, maxPoolSize=2 2016-10-13 15:41:03,081 DEBUG [RS:0;10.22.16.34:59253] executor.ExecutorService(101): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-10.22.16.34:59253, corePoolSize=10, maxPoolSize=10 2016-10-13 15:41:03,081 DEBUG [RS:0;10.22.16.34:59233] executor.ExecutorService(101): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-10.22.16.34:59233, corePoolSize=10, maxPoolSize=10 2016-10-13 15:41:03,081 DEBUG [RS:0;10.22.16.34:59253] executor.ExecutorService(101): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.22.16.34:59253, corePoolSize=3, maxPoolSize=3 2016-10-13 15:41:03,081 DEBUG [RS:0;10.22.16.34:59233] executor.ExecutorService(101): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-10.22.16.34:59233, corePoolSize=3, maxPoolSize=3 2016-10-13 15:41:03,082 DEBUG [RS:0;10.22.16.34:59233] zookeeper.ZKUtil(369): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.22.16.34,59233,1476398451084 2016-10-13 15:41:03,082 DEBUG [RS:0;10.22.16.34:59253] zookeeper.ZKUtil(369): regionserver:59253-0x157c034877e0003, quorum=localhost:55969, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.22.16.34,59233,1476398451084 2016-10-13 15:41:03,083 DEBUG [RS:0;10.22.16.34:59233] zookeeper.ZKUtil(369): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.22.16.34,59253,1476398461576 2016-10-13 15:41:03,083 INFO [RS:0;10.22.16.34:59233] regionserver.ReplicationSourceManager(248): Current list of replicators: [10.22.16.34,59233,1476398451084, 10.22.16.34,59253,1476398461576] other RSs: [10.22.16.34,59233,1476398451084, 10.22.16.34,59253,1476398461576] 2016-10-13 15:41:03,083 DEBUG [RS:0;10.22.16.34:59253] zookeeper.ZKUtil(369): regionserver:59253-0x157c034877e0003, quorum=localhost:55969, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.22.16.34,59253,1476398461576 2016-10-13 15:41:03,083 INFO [RS:0;10.22.16.34:59253] regionserver.ReplicationSourceManager(248): Current list of replicators: [10.22.16.34,59233,1476398451084, 10.22.16.34,59253,1476398461576] other RSs: [10.22.16.34,59233,1476398451084, 10.22.16.34,59253,1476398461576] 2016-10-13 15:41:03,102 INFO [10.22.16.34:59249.activeMasterManager] master.ServerManager(1067): Finished waiting for region servers count to settle; checked in 2, slept for 1052 ms, expecting minimum of 1, maximum of 1, master is running 2016-10-13 15:41:03,103 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,104 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,105 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,108 INFO [RS:0;10.22.16.34:59233] regionserver.HeapMemoryManager(191): Starting HeapMemoryTuner chore. 2016-10-13 15:41:03,108 INFO [SplitLogWorker-10.22.16.34:59233] regionserver.SplitLogWorker(134): SplitLogWorker 10.22.16.34,59233,1476398451084 starting 2016-10-13 15:41:03,108 INFO [SplitLogWorker-10.22.16.34:59253] regionserver.SplitLogWorker(134): SplitLogWorker 10.22.16.34,59253,1476398461576 starting 2016-10-13 15:41:03,108 INFO [RS:0;10.22.16.34:59253] regionserver.HeapMemoryManager(191): Starting HeapMemoryTuner chore. 2016-10-13 15:41:03,108 INFO [RS:0;10.22.16.34:59233] regionserver.HRegionServer(1406): Serving as 10.22.16.34,59233,1476398451084, RpcServer on 10.22.16.34/10.22.16.34:59233, sessionid=0x157c034877e0000 2016-10-13 15:41:03,108 DEBUG [RS:0;10.22.16.34:59233] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2016-10-13 15:41:03,108 INFO [RS:0;10.22.16.34:59253] regionserver.HRegionServer(1406): Serving as 10.22.16.34,59253,1476398461576, RpcServer on 10.22.16.34/10.22.16.34:59253, sessionid=0x157c034877e0003 2016-10-13 15:41:03,108 DEBUG [RS:0;10.22.16.34:59233] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.22.16.34,59233,1476398451084 2016-10-13 15:41:03,108 DEBUG [RS:0;10.22.16.34:59233] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.22.16.34,59233,1476398451084' 2016-10-13 15:41:03,108 DEBUG [RS:0;10.22.16.34:59233] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2016-10-13 15:41:03,108 DEBUG [RS:0;10.22.16.34:59253] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2016-10-13 15:41:03,108 DEBUG [RS:0;10.22.16.34:59253] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager 10.22.16.34,59253,1476398461576 2016-10-13 15:41:03,109 DEBUG [RS:0;10.22.16.34:59253] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.22.16.34,59253,1476398461576' 2016-10-13 15:41:03,109 DEBUG [RS:0;10.22.16.34:59253] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2016-10-13 15:41:03,109 DEBUG [RS:0;10.22.16.34:59233] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2016-10-13 15:41:03,109 DEBUG [RS:0;10.22.16.34:59253] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2016-10-13 15:41:03,109 DEBUG [RS:0;10.22.16.34:59233] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2016-10-13 15:41:03,109 DEBUG [RS:0;10.22.16.34:59233] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2016-10-13 15:41:03,109 DEBUG [RS:0;10.22.16.34:59233] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.22.16.34,59233,1476398451084 2016-10-13 15:41:03,109 DEBUG [RS:0;10.22.16.34:59233] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.22.16.34,59233,1476398451084' 2016-10-13 15:41:03,109 DEBUG [RS:0;10.22.16.34:59253] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2016-10-13 15:41:03,109 DEBUG [RS:0;10.22.16.34:59253] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2016-10-13 15:41:03,109 DEBUG [RS:0;10.22.16.34:59253] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager 10.22.16.34,59253,1476398461576 2016-10-13 15:41:03,109 DEBUG [RS:0;10.22.16.34:59253] procedure.ZKProcedureMemberRpcs(352): Starting procedure member '10.22.16.34,59253,1476398461576' 2016-10-13 15:41:03,109 DEBUG [RS:0;10.22.16.34:59253] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2016-10-13 15:41:03,109 DEBUG [RS:0;10.22.16.34:59233] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2016-10-13 15:41:03,109 DEBUG [RS:0;10.22.16.34:59253] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2016-10-13 15:41:03,110 DEBUG [RS:0;10.22.16.34:59233] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2016-10-13 15:41:03,110 DEBUG [RS:0;10.22.16.34:59253] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2016-10-13 15:41:03,110 INFO [RS:0;10.22.16.34:59253] quotas.RegionServerQuotaManager(53): Quota support disabled 2016-10-13 15:41:03,110 DEBUG [RS:0;10.22.16.34:59233] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2016-10-13 15:41:03,110 INFO [RS:0;10.22.16.34:59233] quotas.RegionServerQuotaManager(53): Quota support disabled 2016-10-13 15:41:03,306 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,508 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,710 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,913 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,913 INFO [10.22.16.34:59249.activeMasterManager] zookeeper.MetaTableLocator(532): Deleting hbase:meta region location in ZooKeeper 2016-10-13 15:41:03,916 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.RecoverableZooKeeper(188): Node /hbase/meta-region-server already deleted, retry=false 2016-10-13 15:41:03,939 DEBUG [region-location-0] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/data/hbase/meta/1588230740/info 2016-10-13 15:41:03,942 DEBUG [10.22.16.34:59249.activeMasterManager] master.AssignmentManager(2464): No previous transition plan found (or ignoring an existing plan) for hbase:meta,,1.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=10.22.16.34,59233,1476398451084; 2 (online=2) available servers, forceNewPlan=false 2016-10-13 15:41:03,942 INFO [10.22.16.34:59249.activeMasterManager] master.AssignmentManager(2110): Setting node as OFFLINED in ZooKeeper for region {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2016-10-13 15:41:03,942 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKAssign(205): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Creating (or updating) unassigned node 1588230740 with OFFLINE state 2016-10-13 15:41:03,943 DEBUG [10.22.16.34:59249.activeMasterManager] master.AssignmentManager(2127): Setting table hbase:meta to ENABLED state. 2016-10-13 15:41:03,943 INFO [10.22.16.34:59249.activeMasterManager] zookeeper.ZKTableStateManager(100): Moving table hbase:meta state from null to ENABLED 2016-10-13 15:41:03,946 INFO [10.22.16.34:59249.activeMasterManager] master.AssignmentManager(2142): Assigning hbase:meta,,1.1588230740 to 10.22.16.34,59233,1476398451084 2016-10-13 15:41:03,946 INFO [10.22.16.34:59249.activeMasterManager] master.RegionStates(1118): Transition {1588230740 state=OFFLINE, ts=1476398463942, server=null} to {1588230740 state=PENDING_OPEN, ts=1476398463946, server=10.22.16.34,59233,1476398451084} 2016-10-13 15:41:03,946 DEBUG [10.22.16.34:59249.activeMasterManager] master.ServerManager(976): New admin connection to 10.22.16.34,59233,1476398451084 2016-10-13 15:41:03,946 DEBUG [10.22.16.34:59249.activeMasterManager] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service AdminService, sasl=false 2016-10-13 15:41:03,946 DEBUG [10.22.16.34:59249.activeMasterManager] ipc.RpcClientImpl$Connection(716): Connecting to /10.22.16.34:59233 2016-10-13 15:41:03,947 DEBUG [RpcServer.listener,port=59233] ipc.RpcServer$Listener(898): RpcServer.listener,port=59233: connection from 10.22.16.34:59267; # active connections: 1 2016-10-13 15:41:03,948 INFO [RpcServer.reader=1,bindAddress=10.22.16.34,port=59233] ipc.RpcServer$Connection(1734): Auth successful for tyu (auth:SIMPLE) 2016-10-13 15:41:03,948 INFO [RpcServer.reader=1,bindAddress=10.22.16.34,port=59233] ipc.RpcServer$Connection(1764): Connection from 10.22.16.34 port: 59267 with version info: version: "1.3.0-SNAPSHOT" url: "git://TYus-MacBook-Pro.local/Users/tyu/1.3-h" revision: "28aaa49c6e39ceba96ebbf26aa400a9852ebdfe6" user: "tyu" date: "Thu Oct 13 15:40:24 PDT 2016" src_checksum: "2ab722632609a1916888f7607a252cd5" version_major: 1 version_minor: 3 2016-10-13 15:41:03,949 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1620): Open hbase:meta,,1.1588230740 2016-10-13 15:41:03,949 DEBUG [RS_OPEN_META-10.22.16.34:59233-0] zookeeper.ZKAssign(805): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Transitioning 1588230740 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2016-10-13 15:41:03,949 DEBUG [10.22.16.34:59249.activeMasterManager] procedure.MasterProcedureScheduler(387): Wake event ProcedureEvent(server crash processing) 2016-10-13 15:41:03,950 INFO [10.22.16.34:59249.activeMasterManager] master.ServerManager(698): AssignmentManager hasn't finished failover cleanup; waiting 2016-10-13 15:41:03,950 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2016-10-13 15:41:03,950 DEBUG [RS_OPEN_META-10.22.16.34:59233-0] zookeeper.ZKAssign(880): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Transitioned node 1588230740 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2016-10-13 15:41:03,950 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,950 INFO [RS_OPEN_META-10.22.16.34:59233-0] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2016-10-13 15:41:03,951 DEBUG [AM.ZK.Worker-pool12-t1] master.AssignmentManager(964): Handling RS_ZK_REGION_OPENING, server=10.22.16.34,59233,1476398451084, region=1588230740, current_state={1588230740 state=PENDING_OPEN, ts=1476398463946, server=10.22.16.34,59233,1476398451084} 2016-10-13 15:41:03,951 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,951 INFO [AM.ZK.Worker-pool12-t1] master.RegionStates(1118): Transition {1588230740 state=PENDING_OPEN, ts=1476398463946, server=10.22.16.34,59233,1476398451084} to {1588230740 state=OPENING, ts=1476398463951, server=10.22.16.34,59233,1476398451084} 2016-10-13 15:41:03,951 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,952 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,952 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,953 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,953 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,953 INFO [RS_OPEN_META-10.22.16.34:59233-0] wal.FSHLog(527): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.22.16.34%2C59233%2C1476398451084.meta, suffix=.meta, logDir=hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/WALs/10.22.16.34,59233,1476398451084, archiveDir=hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/oldWALs 2016-10-13 15:41:03,953 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,954 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,954 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,955 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,956 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,956 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,956 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,957 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,957 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,958 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,959 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,959 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,960 INFO [RS_OPEN_META-10.22.16.34:59233-0] wal.FSHLog(1420): Slow sync cost: 4 ms, current pipeline: [] 2016-10-13 15:41:03,960 INFO [RS_OPEN_META-10.22.16.34:59233-0] wal.FSHLog(893): New WAL /user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/WALs/10.22.16.34,59233,1476398451084/10.22.16.34%2C59233%2C1476398451084.meta.1476398463953.meta 2016-10-13 15:41:03,960 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,960 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,961 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,961 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,961 DEBUG [RS_OPEN_META-10.22.16.34:59233-0] regionserver.HRegion(6662): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2016-10-13 15:41:03,962 DEBUG [RS_OPEN_META-10.22.16.34:59233-0] coprocessor.CoprocessorHost(182): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2016-10-13 15:41:03,962 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,962 DEBUG [RS_OPEN_META-10.22.16.34:59233-0] regionserver.HRegion(8041): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2016-10-13 15:41:03,963 INFO [RS_OPEN_META-10.22.16.34:59233-0] regionserver.RegionCoprocessorHost(367): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2016-10-13 15:41:03,963 DEBUG [RS_OPEN_META-10.22.16.34:59233-0] regionserver.MetricsRegionSourceImpl(74): Creating new MetricsRegionSourceImpl for table meta 1588230740 2016-10-13 15:41:03,963 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,963 DEBUG [RS_OPEN_META-10.22.16.34:59233-0] regionserver.HRegion(740): Instantiated hbase:meta,,1.1588230740 2016-10-13 15:41:03,963 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,964 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,964 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,964 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,965 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,965 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(241): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=1071848, freeSize=1042890456, maxSize=1043962304, heapSize=1071848, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-13 15:41:03,965 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(149): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2016-10-13 15:41:03,965 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,966 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,966 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/data/hbase/meta/1588230740/info 2016-10-13 15:41:03,967 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,967 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,968 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,968 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,968 DEBUG [RS_OPEN_META-10.22.16.34:59233-0] regionserver.HRegion(4045): Found 0 recovered edits file(s) under hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/data/hbase/meta/1588230740 2016-10-13 15:41:03,969 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,969 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,969 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,970 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,970 DEBUG [RS_OPEN_META-10.22.16.34:59233-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2016-10-13 15:41:03,970 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,971 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,971 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,972 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,972 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,972 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,973 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,973 DEBUG [RS_OPEN_META-10.22.16.34:59233-0] wal.WALSplitter(743): Wrote region seqId=hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/data/hbase/meta/1588230740/recovered.edits/3.seqid to file, newSeqId=3, maxSeqId=2 2016-10-13 15:41:03,973 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,974 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,974 INFO [RS_OPEN_META-10.22.16.34:59233-0] regionserver.HRegion(882): Onlined 1588230740; next sequenceid=3 2016-10-13 15:41:03,975 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,975 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,975 DEBUG [RS_OPEN_META-10.22.16.34:59233-0] zookeeper.ZKAssign(643): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Attempting to retransition opening state of node 1588230740 2016-10-13 15:41:03,975 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,976 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1942): Post open deploy tasks for hbase:meta,,1.1588230740 2016-10-13 15:41:03,976 INFO [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(446): Setting hbase:meta region location in ZooKeeper as 10.22.16.34,59233,1476398451084 2016-10-13 15:41:03,976 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,976 DEBUG [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(458): META region location doesn't exist, create it 2016-10-13 15:41:03,976 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKUtil(628): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-10-13 15:41:03,977 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1976): Finished post open deploy task for hbase:meta,,1.1588230740 2016-10-13 15:41:03,977 DEBUG [RS_OPEN_META-10.22.16.34:59233-0] zookeeper.ZKAssign(805): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Transitioning 1588230740 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2016-10-13 15:41:03,978 DEBUG [RS_OPEN_META-10.22.16.34:59233-0] zookeeper.ZKAssign(880): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Transitioned node 1588230740 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2016-10-13 15:41:03,978 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2016-10-13 15:41:03,978 DEBUG [RS_OPEN_META-10.22.16.34:59233-0] coordination.ZkOpenRegionCoordination(91): Transitioned 1588230740 to OPENED in zk on 10.22.16.34,59233,1476398451084 2016-10-13 15:41:03,978 DEBUG [RS_OPEN_META-10.22.16.34:59233-0] handler.OpenRegionHandler(170): Opened hbase:meta,,1.1588230740 on 10.22.16.34,59233,1476398451084 2016-10-13 15:41:03,979 DEBUG [AM.ZK.Worker-pool12-t2] master.AssignmentManager(964): Handling RS_ZK_REGION_OPENED, server=10.22.16.34,59233,1476398451084, region=1588230740, current_state={1588230740 state=OPENING, ts=1476398463951, server=10.22.16.34,59233,1476398451084} 2016-10-13 15:41:03,979 INFO [AM.ZK.Worker-pool12-t2] master.RegionStates(1118): Transition {1588230740 state=OPENING, ts=1476398463951, server=10.22.16.34,59233,1476398451084} to {1588230740 state=OPEN, ts=1476398463979, server=10.22.16.34,59233,1476398451084} 2016-10-13 15:41:03,979 INFO [AM.ZK.Worker-pool12-t2] coordination.ZkOpenRegionCoordination(348): Handling OPENED of 1588230740 from 10.22.16.34,59249,1476398461537; deleting unassigned node 2016-10-13 15:41:03,979 DEBUG [AM.ZK.Worker-pool12-t2] zookeeper.ZKAssign(479): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Deleted unassigned node 1588230740 in expected state RS_ZK_REGION_OPENED 2016-10-13 15:41:03,979 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2016-10-13 15:41:03,980 DEBUG [AM.ZK.Worker-pool12-t3] master.AssignmentManager$4(1355): Znode hbase:meta,,1.1588230740 deleted, state: {1588230740 state=OPEN, ts=1476398463979, server=10.22.16.34,59233,1476398451084} 2016-10-13 15:41:03,980 DEBUG [AM.ZK.Worker-pool12-t3] master.RegionStates(466): Onlined 1588230740 on 10.22.16.34,59233,1476398451084 2016-10-13 15:41:03,981 INFO [10.22.16.34:59249.activeMasterManager] master.HMaster(986): hbase:meta with replicaId 0 assigned=1, rit=false, location=10.22.16.34,59233,1476398451084 2016-10-13 15:41:03,982 DEBUG [hconnection-0x3fb7cc2a-shared--pool16-t1] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-13 15:41:03,982 DEBUG [hconnection-0x3fb7cc2a-shared--pool16-t1] ipc.RpcClientImpl$Connection(716): Connecting to /10.22.16.34:59233 2016-10-13 15:41:03,982 DEBUG [RpcServer.listener,port=59233] ipc.RpcServer$Listener(898): RpcServer.listener,port=59233: connection from 10.22.16.34:59269; # active connections: 2 2016-10-13 15:41:03,983 INFO [RpcServer.reader=2,bindAddress=10.22.16.34,port=59233] ipc.RpcServer$Connection(1734): Auth successful for tyu (auth:SIMPLE) 2016-10-13 15:41:03,983 INFO [RpcServer.reader=2,bindAddress=10.22.16.34,port=59233] ipc.RpcServer$Connection(1764): Connection from 10.22.16.34 port: 59269 with version info: version: "1.3.0-SNAPSHOT" url: "git://TYus-MacBook-Pro.local/Users/tyu/1.3-h" revision: "28aaa49c6e39ceba96ebbf26aa400a9852ebdfe6" user: "tyu" date: "Thu Oct 13 15:40:24 PDT 2016" src_checksum: "2ab722632609a1916888f7607a252cd5" version_major: 1 version_minor: 3 2016-10-13 15:41:03,985 INFO [10.22.16.34:59249.activeMasterManager] hbase.MetaMigrationConvertingToPB(165): hbase:meta doesn't have any entries to update. 2016-10-13 15:41:03,985 INFO [10.22.16.34:59249.activeMasterManager] hbase.MetaMigrationConvertingToPB(131): META already up-to date with PB serialization 2016-10-13 15:41:03,987 DEBUG [10.22.16.34:59249.activeMasterManager] zookeeper.ZKAssign(497): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Deleting any existing unassigned nodes 2016-10-13 15:41:03,988 INFO [10.22.16.34:59249.activeMasterManager] master.AssignmentManager(649): Clean cluster startup. Assigning user regions 2016-10-13 15:41:03,988 INFO [10.22.16.34:59249.activeMasterManager] master.AssignmentManager(507): Joined the cluster in 3ms, failover=false 2016-10-13 15:41:03,989 DEBUG [10.22.16.34:59249.activeMasterManager] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/data/hbase/meta/1588230740/info 2016-10-13 15:41:03,993 INFO [10.22.16.34:59249.activeMasterManager] master.TableNamespaceManager(85): Namespace table not found. Creating... 2016-10-13 15:41:03,993 INFO [10.22.16.34:59249.activeMasterManager] master.HMaster(1582): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2016-10-13 15:41:04,107 DEBUG [10.22.16.34:59249.activeMasterManager] procedure2.ProcedureExecutor(671): Procedure CreateTableProcedure (table=hbase:namespace) id=1 owner=tyu.hfs.1 state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2016-10-13 15:41:04,110 DEBUG [ProcedureExecutor-0] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/hbase:namespace/write-master:592490000000000 2016-10-13 15:41:04,117 INFO [RS:0;10.22.16.34:59233] wal.FSHLog(527): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.22.16.34%2C59233%2C1476398451084, suffix=, logDir=hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/WALs/10.22.16.34,59233,1476398451084, archiveDir=hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/oldWALs 2016-10-13 15:41:04,117 INFO [RS:0;10.22.16.34:59253] wal.FSHLog(527): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=10.22.16.34%2C59253%2C1476398461576, suffix=, logDir=hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/WALs/10.22.16.34,59253,1476398461576, archiveDir=hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/oldWALs 2016-10-13 15:41:04,125 INFO [RS:0;10.22.16.34:59233] wal.FSHLog(1420): Slow sync cost: 5 ms, current pipeline: [] 2016-10-13 15:41:04,125 INFO [RS:0;10.22.16.34:59253] wal.FSHLog(1420): Slow sync cost: 5 ms, current pipeline: [] 2016-10-13 15:41:04,125 INFO [RS:0;10.22.16.34:59233] wal.FSHLog(893): New WAL /user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/WALs/10.22.16.34,59233,1476398451084/10.22.16.34%2C59233%2C1476398451084.1476398464117 2016-10-13 15:41:04,125 INFO [RS:0;10.22.16.34:59253] wal.FSHLog(893): New WAL /user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/WALs/10.22.16.34,59253,1476398461576/10.22.16.34%2C59253%2C1476398461576.1476398464117 2016-10-13 15:41:04,227 INFO [IPC Server handler 5 on 59227] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59228 is added to blk_1073741833_1009{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-db86b596-4d7b-4cf7-9974-02d746fadd02:NORMAL|RBW]]} size 312 2016-10-13 15:41:04,630 DEBUG [ProcedureExecutor-0] util.FSTableDescriptors(660): Wrote descriptor into: hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2016-10-13 15:41:04,631 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6473): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/.tmp Table name == hbase:namespace 2016-10-13 15:41:04,638 INFO [IPC Server handler 5 on 59227] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59228 is added to blk_1073741834_1010{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-c3973975-1bcc-4f64-82f9-cc946695f6b0:NORMAL|RBW]]} size 42 2016-10-13 15:41:05,044 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(740): Instantiated hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342. 2016-10-13 15:41:05,044 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1417): Closing hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342.: disabling compactions & flushes 2016-10-13 15:41:05,045 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1444): Updates disabled for region hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342. 2016-10-13 15:41:05,045 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1548): Closed hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342. 2016-10-13 15:41:05,154 INFO [ProcedureExecutor-0] hbase.MetaTableAccessor(1199): Added 1 2016-10-13 15:41:05,262 INFO [ProcedureExecutor-0] zookeeper.ZKTableStateManager(100): Moving table hbase:namespace state from null to ENABLING 2016-10-13 15:41:05,269 DEBUG [region-location-0] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/data/hbase/namespace/7ab1d0b3b7be8b9a7ce9af61c29d5342/info 2016-10-13 15:41:05,269 INFO [ProcedureExecutor-0] master.AssignmentManager(1623): Assigning 1 region(s) to 10.22.16.34,59233,1476398451084 2016-10-13 15:41:05,270 DEBUG [ProcedureExecutor-0] zookeeper.ZKAssign(174): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Async create of unassigned node 7ab1d0b3b7be8b9a7ce9af61c29d5342 with OFFLINE state 2016-10-13 15:41:05,271 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2016-10-13 15:41:05,271 DEBUG [main-EventThread] master.OfflineCallback(69): rs={7ab1d0b3b7be8b9a7ce9af61c29d5342 state=OFFLINE, ts=1476398465264, server=null}, server=10.22.16.34,59233,1476398451084 2016-10-13 15:41:05,271 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={7ab1d0b3b7be8b9a7ce9af61c29d5342 state=OFFLINE, ts=1476398465264, server=null}, server=10.22.16.34,59233,1476398451084 2016-10-13 15:41:05,275 DEBUG [ProcedureExecutor-0] master.AssignmentManager(1676): 10.22.16.34,59233,1476398451084 unassigned znodes=1 of total=1; oldCounter=0 2016-10-13 15:41:05,275 INFO [ProcedureExecutor-0] master.RegionStates(1118): Transition {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=OFFLINE, ts=1476398465270, server=null} to {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398465275, server=10.22.16.34,59233,1476398451084} 2016-10-13 15:41:05,276 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1620): Open hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342. 2016-10-13 15:41:05,277 ERROR [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.RpcServer(2320): Unexpected throwable object java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:1679) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22737) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:05,278 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 1 service: AdminService methodName: OpenRegion size: 81 connection: 10.22.16.34:59267 java.io.IOException at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2321) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:1679) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22737) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) ... 3 more 2016-10-13 15:41:05,278 INFO [ProcedureExecutor-0] master.AssignmentManager(1789): Unable to communicate with 10.22.16.34,59233,1476398451084 in order to assign regions, java.io.IOException: java.io.IOException at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2321) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:1679) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22737) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) ... 3 more at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:802) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1730) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2839) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2818) at org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:288) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:460) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:59) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:139) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:499) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1062) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:857) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:810) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:496) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2321) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:1679) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22737) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) ... 3 more at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:23623) at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:799) ... 14 more 2016-10-13 15:41:05,280 INFO [ProcedureExecutor-0] zookeeper.ZKTableStateManager(100): Moving table hbase:namespace state from ENABLING to ENABLED 2016-10-13 15:41:05,281 DEBUG [AM.-pool11-t1] master.AssignmentManager(1972): Force region state offline {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398465275, server=10.22.16.34,59233,1476398451084} 2016-10-13 15:41:05,284 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:05,284 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2851): Received CLOSE for the region:7ab1d0b3b7be8b9a7ce9af61c29d5342 , which we are already trying to OPEN. Cancelling OPENING. 2016-10-13 15:41:05,284 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2863): The opening previously in progress has been cancelled by a CLOSE request. 2016-10-13 15:41:05,284 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 2 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was opening but not yet served. Opening is cancelled. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2865) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:05,284 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398465275, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:05,285 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 60000ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was opening but not yet served. Opening is cancelled. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2865) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was opening but not yet served. Opening is cancelled. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2865) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:05,388 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:05,388 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:05,388 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 3 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:05,390 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398465285, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:05,390 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 59895ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:05,493 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:05,493 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:05,493 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 4 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:05,494 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398465390, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:05,495 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 59790ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:05,598 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:05,598 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:05,598 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 5 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:05,599 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398465495, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:05,599 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 59686ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:05,600 DEBUG [ProcedureExecutor-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/hbase:namespace/write-master:592490000000000 2016-10-13 15:41:05,600 DEBUG [ProcedureExecutor-0] procedure2.ProcedureExecutor(872): Procedure completed in 1.5020sec: CreateTableProcedure (table=hbase:namespace) id=1 owner=tyu.hfs.1 state=FINISHED 2016-10-13 15:41:05,702 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:05,702 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:05,702 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 6 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:05,703 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398465599, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:05,703 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 59582ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:05,807 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:05,807 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:05,807 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 7 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:05,808 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398465703, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:05,808 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 59477ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:05,913 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:05,913 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:05,913 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 8 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:05,915 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398465808, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:05,915 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 59370ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:06,018 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:06,019 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:06,019 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 9 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:06,020 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398465915, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:06,020 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 59265ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:06,121 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:06,121 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:06,122 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 10 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:06,122 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398466020, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:06,123 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 59162ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:06,225 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:06,226 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:06,226 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 11 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:06,227 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398466123, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:06,227 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 59058ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:06,329 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:06,329 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:06,330 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 12 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:06,331 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398466227, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:06,331 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 58954ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:06,436 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:06,437 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:06,437 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 13 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:06,438 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398466331, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:06,438 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 58847ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:06,544 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:06,544 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:06,544 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 14 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:06,546 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398466438, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:06,546 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 58739ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:06,647 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:06,647 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:06,647 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 15 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:06,648 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398466546, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:06,648 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 58637ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:06,753 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:06,753 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:06,753 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 16 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:06,754 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398466648, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:06,754 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 58531ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:06,857 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:06,857 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:06,857 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 17 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:06,859 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398466754, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:06,859 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 58426ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:06,960 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:06,960 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:06,961 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 18 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:06,963 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398466859, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:06,963 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 58322ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:07,067 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:07,067 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:07,067 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 19 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:07,068 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398466963, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:07,068 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 58217ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:07,171 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:07,171 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:07,172 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 20 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:07,173 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398467068, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:07,173 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 58112ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:07,276 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:07,277 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:07,277 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 21 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:07,277 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398467173, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:07,278 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 58007ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:07,383 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:07,384 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:07,384 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 22 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:07,385 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398467278, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:07,385 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 57900ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:07,490 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:07,490 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:07,491 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 23 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:07,491 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398467385, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:07,491 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 57794ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:07,594 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:07,594 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:07,594 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 24 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:07,595 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398467491, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:07,595 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 57690ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:07,699 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:07,700 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:07,700 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 25 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:07,701 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398467595, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:07,701 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 57584ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:07,803 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:07,803 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:07,804 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 26 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:07,805 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398467701, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:07,805 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 57480ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:07,907 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:07,907 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:07,907 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 27 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:07,908 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398467805, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:07,908 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 57377ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:08,013 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:08,013 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:08,013 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 28 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:08,015 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398467908, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:08,015 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 57270ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:08,120 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:08,120 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:08,120 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 29 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:08,122 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398468015, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:08,122 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 57163ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:08,227 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:08,227 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:08,227 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 30 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:08,228 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398468122, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:08,229 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 57056ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:08,332 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:08,332 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:08,333 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 31 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:08,333 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398468229, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:08,334 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 56951ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:08,438 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:08,438 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:08,438 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 32 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:08,439 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398468334, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:08,439 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 56846ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:08,545 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:08,545 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:08,545 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 33 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:08,546 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398468439, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:08,547 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 56738ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:08,649 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:08,649 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:08,649 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 34 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:08,651 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398468547, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:08,651 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 56634ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:08,757 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:08,757 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:08,757 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 35 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:08,758 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398468651, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:08,758 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 56527ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:08,864 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:08,864 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:08,864 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 36 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:08,865 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398468758, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:08,865 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 56420ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:08,970 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:08,970 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:08,970 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 37 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:08,972 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398468865, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:08,972 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 56313ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:09,076 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:09,076 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:09,076 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 38 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:09,077 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398468972, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:09,077 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 56208ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:09,181 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:09,181 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:09,182 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 39 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:09,182 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398469077, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:09,182 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 56103ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:09,287 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:09,288 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:09,288 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 40 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:09,288 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398469182, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:09,288 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 55997ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:09,389 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:09,389 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:09,389 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 41 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:09,390 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398469288, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:09,390 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 55895ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:09,472 WARN [HBase-Metrics2-1] impl.MetricsConfig(124): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2016-10-13 15:41:09,477 ERROR [HBase-Metrics2-1] lib.MethodMetric$2(118): Error invoking method getBlocksTotal java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111) at org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144) at org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:387) at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79) at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:195) at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172) at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57) at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:221) at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:96) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:245) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:229) at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:290) at com.sun.proxy.$Proxy12.postStart(Unknown Source) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:185) at org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable.run(JmxCacheBuster.java:109) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3158) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5652) ... 33 more 2016-10-13 15:41:09,490 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2016-10-13 15:41:09,496 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:09,496 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:09,496 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 42 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:09,496 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398469390, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:09,496 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 55789ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:09,601 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:09,601 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:09,601 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 43 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:09,602 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398469496, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:09,603 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 55682ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:09,707 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:09,707 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:09,707 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 44 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:09,707 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398469603, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:09,708 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 55577ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:09,812 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:09,812 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:09,812 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 45 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:09,813 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398469708, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:09,813 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 55472ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:09,916 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:09,917 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:09,917 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 46 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:09,917 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398469813, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:09,917 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 55368ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:10,022 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:10,022 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:10,022 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 47 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:10,023 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398469917, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:10,023 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 55262ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:10,126 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:10,126 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:10,126 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 48 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:10,127 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398470023, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:10,127 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 55158ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:10,232 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:10,232 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:10,232 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 49 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:10,232 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398470127, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:10,233 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 55052ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:10,338 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:10,338 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:10,338 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 50 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:10,339 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398470233, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:10,339 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 54946ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:10,445 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:10,445 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:10,445 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 51 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:10,446 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398470339, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:10,446 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 54839ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:10,549 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:10,549 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:10,549 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 52 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:10,550 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398470446, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:10,550 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 54735ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:10,652 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:10,652 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:10,652 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 53 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:10,653 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398470550, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:10,653 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 54632ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:10,759 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:10,759 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:10,759 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 54 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:10,759 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398470653, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:10,760 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 54525ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:10,861 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:10,861 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:10,861 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 55 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:10,862 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398470760, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:10,862 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 54423ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:10,963 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:10,963 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:10,963 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 56 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:10,964 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398470862, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:10,964 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 54321ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:11,067 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:11,067 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:11,067 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 57 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:11,068 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398470964, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:11,068 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 54217ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:11,169 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:11,169 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:11,169 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 58 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:11,170 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398471068, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:11,170 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 54115ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:11,271 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:11,271 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:11,271 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 59 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:11,272 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398471170, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:11,272 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 54013ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:11,375 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:11,375 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:11,375 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 60 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:11,376 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398471272, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:11,376 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 53909ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:11,477 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:11,477 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:11,477 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 61 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:11,478 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398471376, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:11,478 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 53807ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:11,583 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:11,583 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:11,583 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 62 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:11,583 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398471478, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:11,583 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 53702ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:11,687 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:11,687 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:11,688 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 63 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:11,688 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398471583, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:11,688 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 53597ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:11,790 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:11,790 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:11,790 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 64 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:11,790 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398471688, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:11,790 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 53495ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:11,893 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:11,893 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:11,893 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 65 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:11,894 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398471790, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:11,894 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 53391ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:11,998 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:11,998 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:11,998 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 66 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:11,998 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398471894, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:11,998 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 53287ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:12,103 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:12,103 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:12,103 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 67 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:12,104 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398471998, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:12,104 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 53181ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:12,204 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:12,205 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:12,205 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 68 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:12,205 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398472104, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:12,205 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 53080ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:12,306 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:12,306 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:12,306 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 69 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:12,306 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398472205, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:12,306 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 52979ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:12,407 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:12,407 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:12,407 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 70 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:12,407 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398472306, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:12,407 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 52878ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:12,508 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:12,509 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:12,509 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 71 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:12,509 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398472407, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:12,509 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 52776ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:12,615 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:12,615 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:12,616 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 72 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:12,616 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398472509, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:12,616 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 52669ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:12,721 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:12,721 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:12,721 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 73 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:12,722 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398472616, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:12,722 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 52563ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:12,823 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:12,823 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:12,823 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 74 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:12,824 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398472722, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:12,824 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 52461ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:12,930 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:12,930 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:12,930 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 75 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:12,930 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398472824, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:12,930 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 52355ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:13,031 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:13,032 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:13,032 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 76 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:13,032 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398472930, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:13,032 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 52253ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:13,138 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:13,138 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:13,138 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 77 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:13,138 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398473032, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:13,138 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 52147ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:13,244 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:13,244 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:13,244 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 78 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:13,245 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398473138, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:13,245 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 52040ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:13,351 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:13,351 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:13,351 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 79 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:13,352 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398473245, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:13,352 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 51933ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:13,457 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:13,457 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:13,457 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 80 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:13,457 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398473352, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:13,457 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 51828ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:13,558 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:13,558 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:13,558 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 81 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:13,559 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398473457, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:13,559 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 51726ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:13,663 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:13,663 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:13,663 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 82 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:13,664 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398473559, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:13,664 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 51621ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:13,767 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:13,767 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:13,768 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 83 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:13,768 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398473664, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:13,769 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 51516ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:13,871 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:13,871 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:13,871 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 84 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:13,872 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398473769, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:13,872 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 51413ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:13,973 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:13,973 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:13,973 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 85 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:13,973 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398473872, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:13,973 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 51312ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:14,075 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:14,075 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:14,075 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 86 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:14,076 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398473973, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:14,076 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 51209ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:14,177 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:14,177 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:14,177 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 87 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:14,177 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398474076, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:14,177 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 51108ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:14,279 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:14,279 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:14,279 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 88 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:14,279 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398474177, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:14,279 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 51006ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:14,385 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:14,385 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:14,385 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 89 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:14,386 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398474279, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:14,386 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 50899ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:14,487 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:14,487 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:14,488 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 90 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:14,488 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398474386, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:14,488 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 50797ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:14,594 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:14,594 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:14,594 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 91 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:14,595 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398474488, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:14,595 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 50690ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:14,698 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:14,698 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:14,698 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 92 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:14,699 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398474595, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:14,699 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 50586ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:14,804 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:14,804 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:14,804 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 93 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:14,804 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398474699, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:14,805 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 50481ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:14,910 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:14,910 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:14,911 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 94 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:14,911 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398474804, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:14,911 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 50374ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:15,016 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:15,016 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:15,016 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 95 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:15,017 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398474911, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:15,017 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 50268ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:15,121 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:15,121 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:15,121 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 96 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:15,122 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398475017, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:15,122 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 50163ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:15,226 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:15,227 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:15,227 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 97 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:15,227 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398475122, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:15,227 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 50058ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:15,328 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:15,329 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:15,329 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 98 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:15,329 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398475227, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:15,329 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 49956ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:15,434 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:15,434 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:15,434 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 99 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:15,434 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398475329, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:15,435 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 49850ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:15,538 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:15,538 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:15,538 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 100 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:15,538 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398475435, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:15,538 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 49747ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:15,644 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:15,645 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:15,645 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 101 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:15,645 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398475538, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:15,645 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 49640ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:15,750 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:15,750 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:15,750 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 102 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:15,751 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398475645, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:15,751 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 49534ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:15,856 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:15,856 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:15,856 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 103 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:15,857 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398475751, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:15,857 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 49428ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:15,960 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:15,960 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:15,960 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 104 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:15,961 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398475857, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:15,961 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 49324ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:16,066 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:16,066 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:16,067 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 105 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:16,067 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398475961, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:16,067 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 49218ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:16,171 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:16,171 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:16,171 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 106 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:16,172 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398476067, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:16,172 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 49113ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:16,276 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:16,276 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:16,276 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 107 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:16,276 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398476172, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:16,277 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 49008ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:16,381 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:16,382 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:16,382 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 108 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:16,382 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398476277, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:16,383 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 48902ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:16,488 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:16,488 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:16,488 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 109 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:16,488 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398476383, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:16,488 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 48797ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:16,592 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:16,592 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:16,592 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 110 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:16,593 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398476488, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:16,593 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 48692ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:16,694 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:16,694 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:16,694 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 111 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:16,694 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398476593, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:16,694 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 48591ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:16,798 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:16,798 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:16,798 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 112 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:16,798 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398476694, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:16,798 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 48487ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:16,900 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:16,900 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:16,900 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 113 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:16,900 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398476798, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:16,901 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 48384ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:17,004 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:17,004 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:17,004 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 114 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:17,004 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398476901, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:17,004 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 48281ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:17,105 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:17,105 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:17,105 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 115 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:17,106 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398477004, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:17,106 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 48179ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:17,212 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:17,212 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:17,212 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 116 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:17,213 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398477106, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:17,213 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 48072ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:17,314 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:17,314 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:17,315 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 117 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:17,315 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398477213, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:17,315 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 47970ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:17,417 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:17,417 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:17,417 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 118 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:17,418 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398477315, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:17,418 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 47867ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:17,521 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:17,521 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:17,521 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 119 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:17,521 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398477418, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:17,521 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 47764ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:17,626 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:17,626 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:17,626 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 120 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:17,626 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398477521, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:17,627 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 47658ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:17,732 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:17,732 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:17,732 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 121 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:17,732 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398477627, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:17,733 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 47552ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:17,837 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:17,837 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:17,837 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 122 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:17,838 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398477733, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:17,838 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 47447ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:17,943 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:17,943 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:17,944 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 123 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:17,944 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398477838, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:17,944 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 47341ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:18,047 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:18,047 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:18,047 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 124 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:18,047 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398477944, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:18,047 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 47238ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:18,151 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:18,151 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:18,151 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 125 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:18,151 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398478047, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:18,151 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 47134ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:18,254 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:18,254 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:18,254 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 126 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:18,255 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398478151, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:18,255 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 47030ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:18,359 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:18,359 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:18,359 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 127 service: AdminService methodName: CloseRegion size: 108 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:18,359 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398478255, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:18,359 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 46926ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:18,463 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:18,464 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:18,464 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 128 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:18,464 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398478359, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:18,464 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 46821ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:18,568 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:18,568 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:18,568 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 129 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:18,568 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398478464, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:18,568 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 46717ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:18,671 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:18,672 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:18,672 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 130 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:18,672 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398478568, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:18,672 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 46613ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:18,773 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:18,773 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:18,773 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 131 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:18,774 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398478672, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:18,774 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 46511ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:18,880 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:18,880 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:18,880 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 132 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:18,880 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398478774, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:18,880 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 46405ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:18,981 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:18,981 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:18,982 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 133 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:18,982 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398478880, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:18,982 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 46303ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:19,088 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:19,088 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:19,088 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 134 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:19,088 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398478982, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:19,088 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 46197ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:19,194 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:19,194 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:19,194 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 135 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:19,194 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398479088, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:19,194 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 46091ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:19,300 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:19,300 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:19,300 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 136 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:19,301 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398479194, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:19,301 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 45984ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:19,405 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:19,405 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:19,405 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 137 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:19,406 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398479301, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:19,406 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 45879ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:19,512 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:19,512 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:19,512 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 138 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:19,512 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398479406, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:19,513 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 45772ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:19,614 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:19,614 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:19,614 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 139 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:19,614 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398479513, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:19,614 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 45671ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:19,718 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:19,718 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:19,718 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 140 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:19,718 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398479614, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:19,718 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 45567ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:19,822 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:19,822 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:19,822 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 141 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:19,823 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398479718, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:19,823 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 45462ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:19,925 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:19,925 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:19,925 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 142 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:19,926 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398479823, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:19,926 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 45359ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:20,031 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:20,031 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:20,031 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 143 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:20,032 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398479926, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:20,032 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 45253ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:20,137 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:20,137 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:20,138 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 144 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:20,138 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398480032, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:20,138 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 45147ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:20,243 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:20,243 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:20,243 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 145 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:20,244 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398480138, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:20,244 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 45041ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:20,348 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:20,348 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:20,348 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 146 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:20,349 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398480244, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:20,349 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 44936ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:20,451 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:20,451 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:20,452 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 147 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:20,452 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398480349, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:20,452 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 44833ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:20,555 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:20,555 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:20,555 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 148 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:20,556 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398480452, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:20,556 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 44729ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:20,661 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:20,662 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:20,662 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 149 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:20,662 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398480556, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:20,662 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 44623ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:20,763 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:20,764 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:20,764 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 150 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:20,764 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398480662, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:20,764 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 44521ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:20,865 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:20,865 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:20,865 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 151 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:20,865 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398480764, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:20,866 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 44419ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:20,971 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:20,971 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:20,971 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 152 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:20,972 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398480866, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:20,972 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 44313ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:21,076 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:21,076 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:21,076 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 153 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:21,077 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398480972, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:21,077 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 44208ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:21,181 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:21,181 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:21,181 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 154 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:21,182 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398481077, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:21,182 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 44103ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:21,287 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:21,287 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:21,287 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 155 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:21,288 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398481182, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:21,288 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 43997ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:21,392 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:21,392 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:21,392 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 156 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:21,393 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398481288, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:21,393 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 43892ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:21,496 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:21,496 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:21,496 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 157 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:21,497 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398481393, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:21,497 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 43788ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:21,600 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:21,600 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:21,600 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 158 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:21,600 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398481497, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:21,601 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 43684ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:21,705 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:21,705 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:21,705 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 159 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:21,706 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398481601, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:21,706 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 43579ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:21,807 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:21,807 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:21,808 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 160 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:21,808 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398481706, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:21,808 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 43477ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:21,913 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:21,913 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:21,913 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 161 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:21,914 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398481808, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:21,914 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 43371ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:22,015 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:22,016 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:22,016 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 162 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:22,016 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398481914, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:22,016 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 43269ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:22,121 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:22,121 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:22,122 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 163 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:22,122 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398482016, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:22,122 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 43163ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:22,226 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:22,226 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:22,226 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 164 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:22,227 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398482122, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:22,227 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 43058ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:22,329 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:22,329 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:22,329 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 165 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:22,329 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398482227, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:22,329 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 42956ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:22,430 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:22,431 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:22,431 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 166 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:22,431 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398482329, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:22,431 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 42854ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:22,533 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:22,533 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:22,533 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 167 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:22,534 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398482431, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:22,534 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 42751ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:22,638 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:22,639 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:22,639 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 168 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:22,639 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398482534, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:22,639 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 42646ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:22,741 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:22,741 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:22,742 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 169 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:22,742 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398482639, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:22,742 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 42543ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:22,848 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:22,848 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:22,848 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 170 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:22,848 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398482742, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:22,849 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 42436ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:22,953 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:22,953 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:22,953 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 171 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:22,954 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398482849, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:22,954 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 42331ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:23,056 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:23,056 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:23,056 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 172 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:23,057 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398482954, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:23,057 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 42228ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:23,158 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:23,159 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:23,159 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 173 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:23,159 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398483057, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:23,159 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 42126ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:23,263 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:23,263 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:23,263 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 174 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:23,264 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398483159, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:23,264 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 42021ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:23,368 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:23,368 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:23,369 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 175 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:23,369 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398483264, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:23,369 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 41916ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:23,471 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:23,471 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:23,471 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 176 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:23,472 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398483369, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:23,472 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 41813ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:23,575 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:23,575 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:23,575 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 177 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:23,576 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398483472, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:23,576 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 41709ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:23,681 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:23,681 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:23,681 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 178 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:23,681 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398483576, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:23,681 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 41604ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:23,782 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:23,782 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:23,782 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 179 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:23,783 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398483681, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:23,783 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 41502ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:23,888 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:23,888 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:23,888 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 180 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:23,889 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398483783, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:23,889 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 41396ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:23,992 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:23,992 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:23,993 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 181 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:23,993 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398483889, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:23,993 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 41292ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:24,098 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:24,098 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:24,098 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 182 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:24,098 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398483993, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:24,099 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 41186ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:24,201 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:24,201 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:24,201 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 183 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:24,201 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398484099, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:24,202 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 41083ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:24,304 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:24,304 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:24,304 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 184 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:24,305 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398484202, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:24,305 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 40980ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:24,409 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:24,409 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:24,410 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 185 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:24,410 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398484305, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:24,410 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 40875ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:24,514 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:24,514 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:24,515 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 186 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:24,515 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398484410, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:24,515 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 40770ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:24,620 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:24,620 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:24,621 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 187 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:24,621 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398484515, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:24,621 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 40664ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:24,724 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:24,724 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:24,724 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 188 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:24,725 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398484621, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:24,725 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 40560ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:24,828 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:24,828 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:24,828 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 189 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:24,828 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398484725, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:24,828 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 40457ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:24,929 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:24,930 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:24,930 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 190 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:24,930 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398484828, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:24,930 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 40355ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:25,035 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:25,035 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:25,035 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 191 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:25,036 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398484930, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:25,036 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 40249ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:25,138 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:25,138 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:25,138 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 192 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:25,138 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398485036, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:25,138 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 40147ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:25,243 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:25,243 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:25,243 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 193 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:25,244 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398485138, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:25,244 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 40041ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:25,348 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:25,348 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:25,348 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 194 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:25,348 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398485244, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:25,349 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 39937ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:25,451 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:25,451 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:25,451 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 195 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:25,451 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398485348, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:25,451 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 39834ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:25,555 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:25,555 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:25,555 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 196 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:25,556 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398485451, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:25,556 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 39729ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:25,661 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:25,661 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:25,661 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 197 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:25,661 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398485556, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:25,661 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 39624ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:25,762 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:25,762 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:25,762 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 198 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:25,762 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398485661, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:25,762 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 39523ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:25,868 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:25,868 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:25,868 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 199 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:25,868 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398485762, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:25,868 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 39417ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:25,972 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:25,972 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:25,972 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 200 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:25,972 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398485868, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:25,972 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 39313ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:26,075 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:26,075 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:26,075 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 201 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:26,076 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398485972, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:26,076 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 39209ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:26,178 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:26,178 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:26,178 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 202 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:26,179 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398486076, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:26,179 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 39106ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:26,281 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:26,281 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:26,281 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 203 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:26,281 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398486179, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:26,281 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 39004ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:26,385 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:26,385 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:26,385 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 204 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:26,385 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398486281, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:26,385 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 38900ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:26,488 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:26,488 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:26,488 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 205 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:26,489 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398486385, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:26,489 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 38796ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:26,592 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:26,593 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:26,593 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 206 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:26,593 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398486489, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:26,593 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 38692ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:26,694 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:26,694 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:26,694 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 207 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:26,695 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398486593, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:26,695 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 38590ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:26,796 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:26,796 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:26,796 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 208 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:26,797 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398486695, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:26,797 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 38488ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:26,898 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:26,898 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:26,898 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 209 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:26,898 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398486797, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:26,899 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 38386ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:27,002 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:27,002 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:27,002 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 210 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:27,003 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398486898, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:27,003 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 38282ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:27,108 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:27,108 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:27,108 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 211 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:27,109 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398487003, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:27,109 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 38176ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:27,213 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:27,213 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:27,213 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 212 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:27,214 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398487109, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:27,214 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 38071ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:27,315 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:27,315 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:27,315 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 213 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:27,316 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398487214, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:27,316 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 37969ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:27,421 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:27,421 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:27,421 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 214 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:27,421 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398487316, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:27,421 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 37864ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:27,522 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:27,522 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:27,523 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 215 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:27,523 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398487421, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:27,523 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 37762ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:27,628 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:27,628 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:27,628 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 216 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:27,628 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398487523, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:27,629 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 37656ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:27,732 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:27,732 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:27,732 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 217 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:27,733 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398487629, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:27,733 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 37552ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:27,837 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:27,837 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:27,837 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 218 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:27,838 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398487733, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:27,838 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 37447ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:27,940 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:27,940 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:27,940 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 219 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:27,940 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398487838, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:27,940 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 37345ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:28,044 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:28,044 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:28,044 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 220 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:28,045 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398487940, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:28,045 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 37240ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:28,148 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:28,148 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:28,148 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 221 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:28,148 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398488045, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:28,148 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 37137ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:28,250 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:28,250 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:28,250 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 222 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:28,251 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398488148, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:28,251 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 37034ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:28,355 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:28,355 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:28,355 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 223 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:28,355 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398488251, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:28,356 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 36929ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:28,457 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:28,457 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:28,457 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 224 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:28,458 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398488356, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:28,458 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 36827ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:28,558 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:28,558 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:28,558 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 225 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:28,559 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398488458, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:28,559 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 36726ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:28,662 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:28,662 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:28,662 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 226 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:28,663 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398488559, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:28,663 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 36622ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:28,766 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:28,766 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:28,766 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 227 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:28,767 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398488663, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:28,767 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 36518ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:28,870 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:28,870 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:28,870 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 228 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:28,871 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398488767, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:28,871 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 36414ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:28,971 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:28,971 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:28,972 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 229 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:28,972 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398488871, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:28,972 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 36313ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:29,077 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:29,078 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:29,078 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 230 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:29,078 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398488972, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:29,078 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 36207ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:29,180 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:29,180 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:29,180 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 231 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:29,180 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398489078, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:29,180 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 36105ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:29,285 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:29,285 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:29,285 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 232 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:29,286 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398489180, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:29,286 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 35999ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:29,389 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:29,389 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:29,389 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 233 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:29,390 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398489286, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:29,390 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 35895ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:29,495 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:29,495 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:29,495 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 234 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:29,495 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398489390, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:29,495 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 35790ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:29,600 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:29,600 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:29,600 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 235 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:29,601 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398489495, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:29,601 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 35684ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:29,706 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:29,706 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:29,706 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 236 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:29,706 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398489601, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:29,706 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 35579ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:29,807 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:29,808 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:29,808 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 237 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:29,808 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398489706, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:29,808 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 35477ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:29,912 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:29,913 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:29,913 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 238 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:29,913 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398489808, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:29,913 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 35372ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:30,015 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:30,015 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:30,015 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 239 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:30,015 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398489913, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:30,016 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 35269ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:30,120 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:30,121 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:30,121 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 240 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:30,121 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398490016, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:30,121 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 35164ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:30,226 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:30,226 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:30,226 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 241 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:30,226 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398490121, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:30,227 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 35058ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:30,330 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:30,330 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:30,330 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 242 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:30,331 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398490227, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:30,331 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 34954ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:30,436 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:30,436 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:30,436 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 243 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:30,436 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398490331, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:30,436 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 34849ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:30,538 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:30,538 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:30,539 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 244 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:30,539 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398490436, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:30,539 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 34746ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:30,641 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:30,641 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:30,641 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 245 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:30,641 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398490539, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:30,642 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 34643ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:30,746 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:30,746 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:30,746 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 246 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:30,747 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398490642, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:30,747 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 34538ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:30,849 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:30,850 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:30,850 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 247 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:30,850 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398490747, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:30,850 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 34435ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:30,956 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:30,956 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:30,956 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 248 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:30,957 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398490850, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:30,957 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 34328ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:31,060 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:31,060 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:31,060 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 249 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:31,061 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398490957, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:31,061 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 34224ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:31,166 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:31,167 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:31,167 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 250 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:31,167 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398491061, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:31,167 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 34118ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:31,272 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:31,272 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:31,273 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 251 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:31,273 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398491167, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:31,273 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 34012ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:31,376 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:31,376 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:31,376 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 252 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:31,377 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398491273, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:31,377 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 33908ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:31,481 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:31,481 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:31,482 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 253 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:31,482 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398491377, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:31,482 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 33803ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:31,587 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:31,587 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:31,587 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 254 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:31,588 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398491482, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:31,588 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 33697ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:31,689 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:31,690 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:31,690 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 255 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:31,690 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398491588, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:31,690 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 33595ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:31,791 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:31,791 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:31,791 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 256 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:31,792 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398491690, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:31,792 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 33493ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:31,894 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:31,894 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:31,894 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 257 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:31,895 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398491792, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:31,895 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 33390ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:31,998 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:31,998 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:31,999 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 258 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:31,999 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398491895, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:31,999 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 33286ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:32,048 DEBUG [ProcedureExecutorTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(180): Evict completed procedure: Procedure=CreateTableProcedure (table=hbase:namespace) (id=1, owner=tyu.hfs.1, state=FINISHED, startTime=28sec ago, lastUpdate=28sec ago) 2016-10-13 15:41:32,101 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:32,101 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:32,101 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 259 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:32,101 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398491999, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:32,102 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 33183ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:32,163 INFO [IPC Server handler 1 on 59227] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59228 is added to blk_1073741830_1006{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-c3973975-1bcc-4f64-82f9-cc946695f6b0:NORMAL|RBW]]} size 477 2016-10-13 15:41:32,164 DEBUG [WALProcedureStoreSyncThread] wal.WALProcedureStore(840): Roll new state log: 2 2016-10-13 15:41:32,165 DEBUG [WALProcedureStoreSyncThread] wal.WALProcedureStore(883): Remove all state logs with ID less than 1 2016-10-13 15:41:32,166 INFO [IPC Server handler 9 on 59227] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741830_1006 127.0.0.1:59228 2016-10-13 15:41:32,167 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore(902): Removed log=hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/MasterProcWALs/state-00000000000000000001.log activeLogs=[hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/MasterProcWALs/state-00000000000000000002.log] 2016-10-13 15:41:32,206 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:32,206 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:32,206 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 260 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:32,206 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398492102, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:32,207 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 33078ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:32,310 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:32,310 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:32,310 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 261 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:32,311 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398492207, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:32,311 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 32974ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:32,415 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:32,416 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:32,416 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 262 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:32,416 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398492311, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:32,416 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 32869ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:32,520 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:32,520 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:32,521 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 263 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:32,521 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398492416, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:32,521 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 32764ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:32,627 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:32,627 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:32,627 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 264 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:32,628 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398492521, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:32,628 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 32657ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:32,733 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:32,733 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:32,733 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 265 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:32,734 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398492628, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:32,734 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 32551ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:32,840 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:32,840 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:32,840 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 266 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:32,840 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398492734, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:32,840 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 32445ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:32,943 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:32,943 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:32,944 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 267 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:32,944 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398492840, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:32,944 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 32341ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:33,049 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:33,049 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:33,049 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 268 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:33,050 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398492944, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:33,050 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 32235ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:33,156 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:33,156 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:33,156 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 269 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:33,157 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398493050, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:33,157 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 32128ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:33,263 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:33,263 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:33,263 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 270 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:33,263 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398493157, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:33,264 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 32021ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:33,366 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:33,366 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:33,366 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 271 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:33,366 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398493264, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:33,366 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 31919ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:33,470 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:33,470 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:33,471 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 272 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:33,471 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398493366, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:33,471 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 31814ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:33,573 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:33,573 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:33,573 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 273 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:33,574 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398493471, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:33,574 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 31711ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:33,677 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:33,677 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:33,677 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 274 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:33,678 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398493574, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:33,678 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 31607ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:33,780 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:33,781 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:33,781 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 275 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:33,781 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398493678, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:33,781 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 31504ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:33,887 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:33,887 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:33,887 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 276 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:33,887 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398493781, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:33,887 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 31398ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:33,993 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:33,993 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:33,993 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 277 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:33,993 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398493887, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:33,994 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 31291ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:34,096 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:34,096 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:34,096 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 278 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:34,096 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398493994, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:34,097 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 31188ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:34,198 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:34,198 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:34,198 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 279 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:34,199 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398494097, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:34,199 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 31086ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:34,300 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:34,300 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:34,300 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 280 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:34,301 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398494199, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:34,301 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 30984ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:34,406 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:34,406 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:34,406 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 281 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:34,407 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398494301, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:34,407 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 30878ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:34,510 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:34,510 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:34,510 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 282 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:34,511 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398494407, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:34,511 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 30774ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:34,616 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:34,616 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:34,616 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 283 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:34,617 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398494511, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:34,617 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 30668ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:34,720 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:34,720 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:34,720 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 284 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:34,721 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398494617, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:34,721 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 30564ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:34,824 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:34,824 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:34,824 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 285 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:34,825 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398494721, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:34,825 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 30460ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:34,929 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:34,930 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:34,930 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 286 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:34,930 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398494825, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:34,930 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 30355ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:35,034 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:35,034 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:35,034 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 287 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:35,035 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398494930, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:35,035 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 30250ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:35,140 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:35,140 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:35,140 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 288 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:35,140 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398495035, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:35,141 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 30144ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:35,243 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:35,243 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:35,243 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 289 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:35,244 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398495141, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:35,244 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 30041ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:35,348 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:35,348 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:35,348 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 290 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:35,349 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398495244, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:35,349 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 29936ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:35,452 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:35,452 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:35,452 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 291 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:35,453 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398495349, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:35,453 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 29832ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:35,556 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:35,556 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:35,556 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 292 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:35,557 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398495453, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:35,557 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 29728ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:35,662 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:35,662 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:35,663 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 293 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:35,663 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398495557, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:35,663 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 29622ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:35,767 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:35,768 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:35,768 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 294 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:35,768 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398495663, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:35,768 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 29517ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:35,873 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:35,873 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:35,873 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 295 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:35,873 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398495768, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:35,873 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 29412ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:35,977 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:35,977 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:35,977 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 296 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:35,977 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398495873, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:35,977 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 29308ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:36,081 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:36,082 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:36,082 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 297 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:36,082 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398495977, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:36,082 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 29203ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:36,186 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:36,186 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:36,187 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 298 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:36,187 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398496082, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:36,187 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 29098ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:36,289 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:36,289 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:36,289 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 299 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:36,289 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398496187, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:36,289 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 28996ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:36,395 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:36,395 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:36,395 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 300 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:36,395 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398496289, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:36,395 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 28890ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:36,499 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:36,499 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:36,499 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 301 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:36,499 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398496395, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:36,500 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 28785ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:36,604 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:36,604 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:36,605 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 302 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:36,605 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398496500, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:36,605 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 28680ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:36,706 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:36,706 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:36,706 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 303 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:36,707 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398496605, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:36,707 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 28578ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:36,811 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:36,811 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:36,811 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 304 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:36,811 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398496707, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:36,811 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 28474ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:36,914 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:36,915 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:36,915 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 305 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:36,915 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398496811, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:36,915 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 28370ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:37,020 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:37,020 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:37,020 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 306 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:37,020 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398496915, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:37,020 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 28265ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:37,125 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:37,125 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:37,126 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 307 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:37,126 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398497020, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:37,126 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 28159ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:37,232 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:37,232 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:37,232 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 308 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:37,232 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398497126, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:37,232 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 28053ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:37,336 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:37,337 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:37,337 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 309 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:37,337 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398497232, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:37,337 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 27948ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:37,439 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:37,440 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:37,440 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 310 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:37,440 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398497337, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:37,440 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 27845ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:37,545 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:37,546 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:37,546 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 311 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:37,546 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398497440, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:37,546 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 27739ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:37,649 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:37,649 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:37,649 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 312 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:37,650 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398497546, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:37,650 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 27635ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:37,755 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:37,756 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:37,756 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 313 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:37,756 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398497650, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:37,756 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 27529ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:37,861 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:37,861 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:37,862 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 314 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:37,862 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398497756, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:37,862 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 27423ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:37,965 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:37,966 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:37,966 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 315 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:37,966 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398497862, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:37,966 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 27319ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:38,070 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:38,070 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:38,070 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 316 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:38,071 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398497966, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:38,071 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 27214ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:38,175 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:38,176 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:38,176 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 317 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:38,176 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398498071, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:38,176 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 27109ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:38,278 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:38,278 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:38,278 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 318 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:38,279 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398498176, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:38,279 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 27006ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:38,384 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:38,384 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:38,384 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 319 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:38,385 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398498279, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:38,385 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 26900ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:38,489 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:38,489 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:38,489 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 320 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:38,490 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398498385, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:38,490 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 26795ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:38,595 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:38,595 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:38,595 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 321 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:38,596 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398498490, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:38,596 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 26689ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:38,700 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:38,700 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:38,700 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 322 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:38,701 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398498596, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:38,701 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 26584ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:38,806 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:38,806 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:38,806 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 323 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:38,807 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398498701, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:38,807 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 26478ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:38,912 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:38,912 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:38,913 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 324 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:38,913 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398498807, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:38,913 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 26372ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:39,014 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:39,015 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:39,015 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 325 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:39,015 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398498913, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:39,015 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 26270ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:39,120 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:39,120 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:39,120 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 326 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:39,120 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398499015, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:39,120 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 26165ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:39,225 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:39,225 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:39,225 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 327 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:39,226 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398499120, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:39,226 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 26059ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:39,330 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:39,331 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:39,331 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 328 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:39,331 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398499226, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:39,331 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 25954ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:39,436 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:39,437 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:39,437 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 329 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:39,437 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398499331, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:39,437 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 25848ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:39,539 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:39,539 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:39,540 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 330 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:39,540 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398499437, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:39,540 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 25745ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:39,645 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:39,645 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:39,646 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 331 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:39,646 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398499540, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:39,646 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 25639ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:39,747 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:39,747 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:39,747 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 332 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:39,748 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398499646, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:39,748 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 25537ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:39,853 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:39,853 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:39,853 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 333 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:39,854 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398499748, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:39,854 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 25431ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:39,957 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:39,957 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:39,957 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 334 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:39,958 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398499854, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:39,958 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 25327ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:40,064 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:40,064 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:40,064 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 335 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:40,065 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398499958, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:40,065 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 25220ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:40,170 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:40,170 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:40,170 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 336 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:40,171 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398500065, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:40,171 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 25114ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:40,276 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:40,276 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:40,276 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 337 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:40,276 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398500171, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:40,276 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 25009ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:40,378 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:40,378 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:40,378 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 338 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:40,378 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398500276, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:40,378 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 24907ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:40,479 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:40,479 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:40,480 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 339 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:40,480 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398500378, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:40,480 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 24805ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:40,584 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:40,584 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:40,584 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 340 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:40,585 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398500480, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:40,585 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 24700ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:40,690 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:40,690 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:40,690 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 341 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:40,691 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398500585, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:40,691 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 24594ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:40,795 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:40,795 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:40,795 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 342 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:40,796 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398500691, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:40,796 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 24489ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:40,900 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:40,900 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:40,901 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 343 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:40,901 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398500796, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:40,901 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 24384ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:41,003 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:41,003 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:41,003 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 344 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:41,004 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398500901, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:41,004 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 24281ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:41,105 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:41,105 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:41,105 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 345 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:41,106 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398501004, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:41,106 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 24179ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:41,207 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:41,207 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:41,208 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 346 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:41,208 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398501106, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:41,208 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 24077ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:41,312 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:41,312 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:41,312 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 347 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:41,313 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398501208, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:41,313 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 23972ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:41,415 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:41,415 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:41,415 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 348 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:41,416 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398501313, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:41,416 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 23869ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:41,520 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:41,520 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:41,520 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 349 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:41,521 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398501416, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:41,521 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 23764ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:41,626 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:41,627 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:41,627 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 350 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:41,627 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398501521, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:41,627 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 23658ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:41,728 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:41,728 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:41,728 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 351 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:41,729 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398501627, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:41,729 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 23556ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:41,831 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:41,831 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:41,832 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 352 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:41,832 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398501729, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:41,832 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 23453ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:41,937 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:41,937 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:41,937 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 353 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:41,937 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398501832, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:41,937 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 23348ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:42,041 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:42,041 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:42,041 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 354 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:42,042 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398501937, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:42,042 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 23243ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:42,147 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:42,147 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:42,147 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 355 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:42,148 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398502042, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:42,148 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 23137ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:42,250 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:42,250 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:42,250 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 356 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:42,250 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398502148, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:42,250 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 23035ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:42,353 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:42,353 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:42,353 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 357 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:42,354 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398502250, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:42,354 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 22931ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:42,459 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:42,460 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:42,460 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 358 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:42,460 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398502354, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:42,460 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 22825ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:42,566 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:42,566 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:42,566 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 359 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:42,567 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398502460, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:42,567 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 22718ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:42,670 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:42,670 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:42,670 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 360 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:42,671 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398502567, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:42,671 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 22614ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:42,774 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:42,774 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:42,774 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 361 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:42,775 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398502671, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:42,775 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 22510ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:42,880 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:42,880 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:42,880 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 362 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:42,881 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398502775, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:42,881 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 22404ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:42,986 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:42,987 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:42,987 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 363 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:42,987 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398502881, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:42,987 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 22298ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:43,091 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:43,091 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:43,091 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 364 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:43,092 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398502987, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:43,092 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 22193ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:43,193 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:43,193 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:43,193 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 365 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:43,193 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398503092, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:43,194 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 22091ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:43,295 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:43,295 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:43,295 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 366 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:43,295 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398503194, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:43,295 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 21990ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:43,399 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:43,399 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:43,399 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 367 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:43,400 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398503295, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:43,400 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 21885ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:43,503 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:43,503 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:43,503 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 368 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:43,504 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398503400, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:43,504 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 21781ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:43,607 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:43,607 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:43,607 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 369 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:43,608 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398503504, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:43,608 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 21677ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:43,711 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:43,711 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:43,711 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 370 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:43,712 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398503608, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:43,712 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 21573ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:43,815 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:43,815 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:43,815 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 371 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:43,816 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398503712, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:43,816 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 21469ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:43,921 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:43,921 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:43,921 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 372 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:43,922 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398503816, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:43,922 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 21363ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:44,024 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:44,024 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:44,024 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 373 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:44,025 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398503922, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:44,025 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 21260ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:44,127 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:44,127 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:44,127 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 374 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:44,127 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398504025, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:44,128 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 21157ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:44,233 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:44,233 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:44,233 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 375 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:44,234 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398504128, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:44,234 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 21051ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:44,337 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:44,338 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:44,338 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 376 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:44,338 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398504234, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:44,338 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 20947ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:44,441 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:44,441 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:44,441 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 377 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:44,442 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398504338, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:44,442 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 20843ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:44,543 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:44,543 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:44,543 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 378 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:44,544 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398504442, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:44,544 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 20741ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:44,645 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:44,645 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:44,645 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 379 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:44,646 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398504544, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:44,646 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 20639ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:44,747 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:44,747 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:44,747 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 380 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:44,747 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398504646, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:44,747 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 20538ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:44,850 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:44,850 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:44,850 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 381 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:44,850 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398504747, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:44,851 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 20434ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:44,954 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:44,954 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:44,954 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 382 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:44,955 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398504851, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:44,955 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 20330ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:45,058 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:45,058 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:45,058 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 383 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:45,059 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398504955, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:45,059 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 20226ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:45,162 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:45,162 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:45,162 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 384 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:45,163 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398505059, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:45,163 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 20122ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:45,265 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:45,266 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:45,266 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 385 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:45,266 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398505163, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:45,266 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 20019ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:45,370 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:45,370 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:45,370 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 386 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:45,370 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398505266, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:45,370 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 19915ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:45,472 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:45,472 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:45,472 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 387 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:45,472 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398505370, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:45,472 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 19813ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:45,573 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:45,573 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:45,573 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 388 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:45,573 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398505472, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:45,573 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 19712ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:45,675 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:45,675 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:45,675 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 389 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:45,675 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398505573, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:45,675 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 19610ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:45,781 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:45,781 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:45,781 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 390 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:45,781 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398505675, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:45,781 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 19504ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:45,886 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:45,886 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:45,886 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 391 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:45,887 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398505781, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:45,887 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 19398ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:45,990 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:45,990 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:45,990 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 392 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:45,990 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398505887, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:45,990 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 19295ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:46,095 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:46,095 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:46,095 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 393 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:46,095 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398505990, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:46,095 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 19190ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:46,199 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:46,199 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:46,199 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 394 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:46,199 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398506095, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:46,199 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 19086ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:46,305 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:46,305 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:46,305 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 395 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:46,305 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398506199, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:46,305 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 18980ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:46,410 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:46,410 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:46,410 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 396 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:46,410 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398506305, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:46,411 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 18874ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:46,511 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:46,511 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:46,511 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 397 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:46,512 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398506411, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:46,512 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 18773ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:46,612 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:46,612 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:46,613 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 398 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:46,613 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398506512, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:46,613 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 18672ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:46,718 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:46,718 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:46,718 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 399 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:46,718 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398506613, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:46,718 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 18567ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:46,820 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:46,820 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:46,820 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 400 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:46,820 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398506718, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:46,821 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 18464ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:46,924 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:46,924 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:46,924 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 401 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:46,925 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398506821, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:46,925 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 18360ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:47,026 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:47,026 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:47,026 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 402 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:47,027 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398506925, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:47,027 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 18258ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:47,128 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:47,128 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:47,128 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 403 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:47,128 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398507027, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:47,129 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 18156ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:47,232 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:47,232 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:47,233 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 404 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:47,233 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398507129, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:47,233 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 18052ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:47,336 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:47,336 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:47,336 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 405 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:47,337 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398507233, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:47,337 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 17948ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:47,440 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:47,440 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:47,440 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 406 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:47,441 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398507337, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:47,441 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 17844ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:47,543 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:47,543 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:47,543 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 407 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:47,543 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398507441, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:47,543 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 17742ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:47,647 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:47,647 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:47,647 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 408 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:47,648 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398507543, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:47,648 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 17637ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:47,753 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:47,753 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:47,753 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 409 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:47,753 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398507648, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:47,753 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 17532ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:47,858 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:47,858 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:47,858 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 410 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:47,859 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398507753, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:47,859 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 17426ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:48,004 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:48,004 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:48,004 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 411 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:48,005 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398507859, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:48,005 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 17280ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:48,108 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:48,108 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:48,108 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 412 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:48,109 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398508005, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:48,109 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 17176ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:48,210 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:48,210 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:48,211 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 413 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:48,211 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398508109, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:48,211 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 17074ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:48,312 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:48,312 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:48,312 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 414 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:48,312 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398508211, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:48,312 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 16973ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:48,413 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:48,413 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:48,413 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 415 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:48,413 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398508312, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:48,413 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 16872ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:48,514 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:48,514 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:48,514 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 416 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:48,514 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398508413, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:48,515 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 16770ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:48,620 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:48,620 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:48,620 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 417 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:48,620 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398508515, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:48,620 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 16665ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:48,725 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:48,725 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:48,726 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 418 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:48,726 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398508620, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:48,726 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 16559ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:48,829 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:48,829 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:48,829 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 419 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:48,829 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398508726, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:48,829 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 16456ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:48,930 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:48,930 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:48,930 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 420 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:48,930 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398508829, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:48,930 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 16355ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:49,036 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:49,036 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:49,036 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 421 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:49,036 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398508930, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:49,036 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 16249ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:49,137 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:49,137 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:49,137 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 422 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:49,138 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398509036, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:49,138 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 16147ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:49,239 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:49,239 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:49,239 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 423 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:49,239 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398509138, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:49,239 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 16046ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:49,342 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:49,342 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:49,342 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 424 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:49,342 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398509239, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:49,342 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 15943ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:49,446 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:49,446 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:49,446 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 425 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:49,447 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398509342, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:49,447 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 15838ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:49,548 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:49,549 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:49,549 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 426 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:49,549 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398509447, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:49,549 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 15736ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:49,653 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:49,653 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:49,653 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 427 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:49,653 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398509549, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:49,653 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 15632ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:49,755 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:49,755 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:49,755 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 428 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:49,755 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398509653, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:49,755 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 15530ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:49,858 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:49,858 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:49,858 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 429 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:49,858 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398509755, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:49,859 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 15426ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:49,963 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:49,963 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:49,963 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 430 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:49,963 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398509859, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:49,963 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 15322ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:50,065 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:50,065 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:50,065 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 431 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:50,066 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398509963, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:50,066 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 15219ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:50,169 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:50,169 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:50,170 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 432 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:50,170 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398510066, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:50,170 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 15115ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:50,274 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:50,274 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:50,274 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 433 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:50,274 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398510170, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:50,274 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 15011ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:50,378 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:50,378 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:50,379 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 434 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:50,379 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398510274, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:50,379 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 14906ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:50,479 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:50,480 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:50,480 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 435 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:50,480 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398510379, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:50,480 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 14805ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:50,585 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:50,585 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:50,585 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 436 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:50,585 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398510480, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:50,585 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 14700ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:50,686 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:50,686 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:50,686 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 437 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:50,686 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398510585, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:50,686 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 14599ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:50,791 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:50,791 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:50,792 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 438 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:50,792 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398510686, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:50,792 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 14493ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:50,897 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:50,897 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:50,897 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 439 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:50,897 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398510792, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:50,897 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 14388ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:50,999 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:50,999 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:50,999 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 440 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:51,000 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398510897, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:51,000 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 14285ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:51,101 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:51,101 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:51,101 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 441 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:51,102 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398511000, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:51,102 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 14183ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:51,205 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:51,205 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:51,205 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 442 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:51,205 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398511102, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:51,205 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 14080ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:51,308 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:51,308 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:51,308 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 443 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:51,308 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398511205, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:51,309 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 13976ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:51,410 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:51,410 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:51,411 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 444 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:51,411 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398511309, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:51,411 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 13874ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:51,514 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:51,514 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:51,514 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 445 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:51,514 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398511411, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:51,514 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 13771ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:51,620 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:51,620 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:51,620 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 446 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:51,621 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398511514, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:51,621 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 13664ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:51,721 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:51,722 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:51,722 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 447 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:51,722 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398511621, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:51,722 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 13563ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:51,824 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:51,825 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:51,825 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 448 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:51,825 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398511722, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:51,825 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 13460ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:51,927 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:51,927 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:51,927 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 449 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:51,927 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398511825, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:51,927 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 13358ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:52,030 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:52,030 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:52,030 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 450 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:52,030 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398511927, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:52,030 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 13255ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:52,136 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:52,136 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:52,136 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 451 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:52,136 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398512030, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:52,136 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 13149ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:52,242 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:52,242 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:52,242 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 452 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:52,242 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398512136, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:52,242 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 13043ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:52,345 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:52,345 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:52,346 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 453 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:52,346 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398512242, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:52,346 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 12939ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:52,447 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:52,447 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:52,447 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 454 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:52,447 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398512346, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:52,447 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 12838ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:52,548 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:52,548 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:52,548 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 455 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:52,548 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398512447, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:52,549 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 12736ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:52,651 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:52,651 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:52,651 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 456 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:52,652 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398512549, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:52,652 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 12633ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:52,754 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:52,754 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:52,754 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 457 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:52,755 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398512652, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:52,755 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 12530ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:52,858 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:52,858 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:52,858 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 458 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:52,858 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398512755, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:52,859 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 12426ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:52,959 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:52,959 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:52,959 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 459 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:52,960 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398512859, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:52,960 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 12325ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:53,064 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:53,064 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:53,064 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 460 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:53,065 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398512960, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:53,065 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 12220ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:53,169 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:53,169 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:53,169 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 461 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:53,170 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398513065, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:53,170 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 12115ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:53,275 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:53,275 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:53,275 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 462 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:53,276 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398513170, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:53,276 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 12009ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:53,380 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:53,380 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:53,380 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 463 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:53,381 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398513276, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:53,381 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 11904ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:53,486 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:53,486 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:53,486 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 464 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:53,486 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398513381, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:53,486 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 11799ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:53,589 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:53,590 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:53,590 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 465 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:53,590 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398513486, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:53,590 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 11695ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:53,691 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:53,691 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:53,691 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 466 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:53,692 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398513590, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:53,692 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 11593ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:53,792 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:53,793 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:53,793 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 467 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:53,793 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398513692, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:53,793 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 11492ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:53,894 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:53,895 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:53,895 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 468 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:53,895 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398513793, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:53,895 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 11390ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:53,998 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:53,998 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:53,998 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 469 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:53,998 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398513895, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:53,998 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 11287ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:54,100 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:54,100 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:54,100 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 470 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:54,100 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398513998, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:54,100 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 11185ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:54,201 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:54,201 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:54,201 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 471 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:54,201 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398514100, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:54,201 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 11084ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:54,303 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:54,303 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:54,303 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 472 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:54,304 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398514201, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:54,304 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 10981ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:54,408 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:54,409 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:54,409 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 473 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:54,409 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398514304, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:54,409 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 10876ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:54,512 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:54,512 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:54,512 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 474 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:54,513 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398514409, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:54,513 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 10772ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:54,618 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:54,618 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:54,618 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 475 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:54,618 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398514513, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:54,618 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 10667ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:54,719 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:54,719 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:54,719 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 476 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:54,720 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398514618, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:54,720 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 10565ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:54,825 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:54,826 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:54,826 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 477 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:54,826 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398514720, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:54,826 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 10459ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:54,928 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:54,928 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:54,928 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 478 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:54,928 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398514826, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:54,928 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 10357ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:55,034 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:55,034 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:55,034 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 479 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:55,035 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398514928, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:55,035 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 10250ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:55,140 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:55,140 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:55,140 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 480 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:55,140 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398515035, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:55,141 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 10144ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:55,244 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:55,244 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:55,244 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 481 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:55,244 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398515141, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:55,244 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 10041ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:55,346 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:55,346 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:55,346 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 482 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:55,346 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398515244, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:55,346 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 9939ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:55,449 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:55,449 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:55,449 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 483 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:55,449 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398515346, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:55,449 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 9836ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:55,554 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:55,554 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:55,554 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 484 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:55,555 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398515449, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:55,555 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 9730ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:55,659 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:55,659 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:55,659 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 485 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:55,659 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398515555, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:55,659 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 9626ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:55,765 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:55,765 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:55,765 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 486 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:55,765 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398515659, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:55,765 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 9520ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:55,869 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:55,869 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:55,869 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 487 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:55,870 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398515765, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:55,870 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 9415ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:55,972 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:55,972 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:55,972 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 488 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:55,972 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398515870, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:55,973 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 9312ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:56,075 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:56,076 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:56,076 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 489 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:56,076 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398515973, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:56,076 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 9209ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:56,179 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:56,179 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:56,179 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 490 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:56,179 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398516076, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:56,179 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 9106ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:56,284 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:56,284 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:56,284 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 491 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:56,284 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398516179, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:56,284 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 9001ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:56,388 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:56,389 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:56,389 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 492 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:56,389 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398516284, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:56,389 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 8896ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:56,492 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:56,492 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:56,492 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 493 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:56,492 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398516389, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:56,492 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 8793ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:56,594 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:56,594 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:56,595 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 494 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:56,595 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398516492, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:56,595 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 8690ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:56,699 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:56,699 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:56,699 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 495 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:56,700 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398516595, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:56,700 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 8585ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:56,804 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:56,804 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:56,804 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 496 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:56,805 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398516700, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:56,805 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 8480ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:56,909 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:56,909 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:56,909 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 497 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:56,909 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398516805, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:56,909 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 8376ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:57,014 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:57,014 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:57,014 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 498 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:57,015 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398516909, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:57,015 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 8270ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:57,119 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:57,119 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:57,119 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 499 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:57,119 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398517015, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:57,119 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 8166ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:57,224 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:57,224 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:57,224 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 500 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:57,224 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398517119, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:57,224 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 8061ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:57,325 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:57,326 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:57,326 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 501 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:57,326 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398517224, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:57,326 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 7959ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:57,431 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:57,431 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:57,432 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 502 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:57,432 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398517326, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:57,432 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 7853ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:57,535 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:57,536 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:57,536 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 503 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:57,536 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398517432, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:57,536 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 7749ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:57,638 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:57,638 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:57,638 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 504 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:57,638 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398517536, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:57,639 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 7646ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:57,742 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:57,742 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:57,742 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 505 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:57,743 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398517639, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:57,743 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 7542ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:57,848 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:57,848 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:57,848 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 506 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:57,849 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398517743, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:57,849 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 7436ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:57,950 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:57,950 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:57,950 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 507 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:57,951 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398517849, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:57,951 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 7334ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:58,056 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:58,056 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:58,056 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 508 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:58,057 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398517951, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:58,057 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 7228ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:58,159 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:58,159 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:58,159 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 509 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:58,159 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398518057, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:58,160 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 7125ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:58,264 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:58,264 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:58,264 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 510 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:58,265 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398518159, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:58,265 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 7020ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:58,369 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:58,369 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:58,369 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 511 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:58,369 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398518265, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:58,369 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 6916ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:58,475 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:58,475 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:58,475 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 512 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:58,475 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398518369, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:58,475 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 6810ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:58,580 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:58,580 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:58,580 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 513 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:58,580 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398518475, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:58,580 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 6705ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:58,685 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:58,686 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:58,686 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 514 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:58,686 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398518580, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:58,686 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 6599ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:58,788 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:58,788 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:58,788 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 515 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:58,788 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398518686, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:58,788 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 6497ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:58,893 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:58,893 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:58,893 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 516 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:58,894 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398518788, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:58,894 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 6391ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:58,996 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:58,996 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:58,996 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 517 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:58,997 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398518894, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:58,997 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 6288ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:59,099 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:59,099 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:59,099 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 518 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:59,100 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398518997, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:59,100 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 6185ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:59,204 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:59,204 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:59,204 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 519 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:59,205 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398519100, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:59,205 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 6080ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:59,309 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:59,309 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:59,309 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 520 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:59,309 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398519205, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:59,309 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 5976ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:59,414 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:59,415 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:59,415 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 521 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:59,415 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398519309, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:59,415 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 5870ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:59,519 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:59,519 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:59,519 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 522 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:59,520 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398519415, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:59,520 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 5765ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:59,624 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:59,624 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:59,624 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 523 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:59,625 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398519520, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:59,625 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 5660ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:59,727 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:59,727 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:59,727 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 524 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:59,727 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398519625, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:59,728 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 5557ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:59,831 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:59,831 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:59,831 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 525 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:59,832 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398519728, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:59,832 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 5453ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:41:59,936 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:41:59,936 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:41:59,936 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 526 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:41:59,936 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398519832, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:41:59,936 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 5349ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:00,037 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:00,037 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:00,037 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 527 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:00,037 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398519936, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:00,038 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 5247ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:00,143 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:00,143 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:00,143 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 528 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:00,143 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398520038, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:00,144 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 5141ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:00,248 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:00,249 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:00,249 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 529 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:00,249 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398520144, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:00,249 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 5036ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:00,350 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:00,350 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:00,350 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 530 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:00,350 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398520249, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:00,350 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 4935ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:00,453 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:00,453 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:00,453 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 531 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:00,454 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398520350, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:00,454 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 4831ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:00,555 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:00,555 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:00,555 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 532 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:00,555 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398520454, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:00,555 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 4730ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:00,659 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:00,660 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:00,660 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 533 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:00,660 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398520555, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:00,660 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 4625ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:00,764 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:00,764 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:00,764 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 534 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:00,765 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398520660, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:00,765 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 4520ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:00,870 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:00,870 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:00,870 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 535 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:00,870 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398520765, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:00,870 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 4415ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:00,973 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:00,974 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:00,974 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 536 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:00,974 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398520870, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:00,974 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 4311ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:01,076 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:01,076 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:01,076 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 537 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:01,076 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398520974, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:01,077 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 4208ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:01,182 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:01,182 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:01,182 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 538 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:01,183 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398521077, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:01,183 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 4102ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:01,286 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:01,287 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:01,287 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 539 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:01,287 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398521183, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:01,287 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 3998ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:01,393 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:01,393 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:01,393 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 540 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:01,393 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398521287, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:01,393 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 3892ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:01,496 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:01,496 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:01,496 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 541 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:01,497 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398521393, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:01,497 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 3788ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:01,602 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:01,603 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:01,603 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 542 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:01,603 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398521497, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:01,603 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 3682ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:01,705 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:01,705 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:01,705 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 543 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:01,705 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398521603, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:01,706 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 3579ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:01,806 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:01,807 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:01,807 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 544 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:01,807 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398521706, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:01,807 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 3478ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:01,912 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:01,913 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:01,913 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 545 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:01,913 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398521807, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:01,913 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 3372ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:02,017 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:02,017 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:02,017 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 546 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:02,017 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398521913, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:02,017 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 3268ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:02,121 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:02,121 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:02,121 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 547 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:02,121 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398522017, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:02,121 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 3164ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:02,225 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:02,225 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:02,225 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 548 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:02,226 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398522121, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:02,226 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 3059ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:02,327 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:02,327 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:02,328 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 549 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:02,328 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398522226, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:02,328 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 2957ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:02,433 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:02,433 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:02,433 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 550 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:02,434 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398522328, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:02,434 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 2851ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:02,535 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:02,536 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:02,536 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 551 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:02,536 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398522434, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:02,536 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 2749ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:02,638 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:02,638 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:02,638 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 552 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:02,639 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398522536, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:02,639 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 2646ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:02,744 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:02,744 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:02,744 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 553 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:02,745 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398522639, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:02,745 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 2540ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:02,850 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:02,850 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:02,850 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 554 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:02,850 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398522745, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:02,850 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 2435ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:02,954 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:02,954 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:02,955 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 555 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:02,955 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398522850, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:02,955 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 2330ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:03,060 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:03,060 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:03,060 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 556 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:03,060 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398522955, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:03,061 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 2224ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:03,164 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:03,164 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:03,164 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 557 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:03,164 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398523061, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:03,164 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 2121ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:03,269 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:03,269 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:03,269 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 558 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:03,270 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398523164, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:03,270 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 2015ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:03,373 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:03,374 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:03,374 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 559 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:03,374 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398523270, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:03,374 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 1911ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:03,477 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:03,477 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:03,477 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 560 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:03,478 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398523374, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:03,478 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 1807ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:03,584 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:03,584 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:03,584 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 561 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:03,584 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398523478, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:03,585 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 1700ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:03,685 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:03,686 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:03,686 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 562 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:03,686 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398523585, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:03,686 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 1599ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:03,788 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:03,789 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:03,789 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 563 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:03,789 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398523686, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:03,789 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 1496ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:03,892 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:03,892 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:03,892 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 564 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:03,892 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398523789, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:03,893 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 1392ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:03,991 DEBUG [region-location-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/data/hbase/meta/1588230740/info 2016-10-13 15:42:03,993 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:03,993 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:03,993 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 565 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:03,994 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398523893, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:03,994 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 1291ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:04,099 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:04,100 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:04,100 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 566 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:04,100 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398523994, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:04,100 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 1185ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:04,205 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:04,205 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:04,205 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 567 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:04,206 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398524100, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:04,206 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 1079ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:04,310 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:04,310 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:04,310 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 568 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:04,311 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398524206, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:04,311 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 974ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:04,414 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:04,414 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:04,414 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 569 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:04,414 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398524311, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:04,414 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 871ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:04,519 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:04,519 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:04,519 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 570 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:04,520 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398524414, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:04,520 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 765ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:04,621 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:04,621 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:04,621 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 571 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:04,622 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398524520, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:04,622 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 663ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:04,727 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:04,727 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:04,727 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 572 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:04,727 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398524622, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:04,727 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 558ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:04,833 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:04,833 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:04,833 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 573 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:04,834 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398524727, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:04,834 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 451ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:04,935 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:04,936 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:04,936 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 574 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:04,936 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398524834, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:04,936 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 349ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:05,042 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:05,042 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:05,042 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 575 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:05,042 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398524936, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:05,042 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 243ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:05,147 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:05,147 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:05,147 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 576 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:05,147 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398525042, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:05,148 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 137ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:05,250 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:05,250 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:05,250 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 577 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:05,251 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398525148, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:05,251 DEBUG [AM.-pool11-t1] master.AssignmentManager(1922): Region is already in transition; waiting up to 34ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:05,353 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:05,353 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:05,353 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 578 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:05,354 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398525251, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:05,354 INFO [AM.-pool11-t1] master.AssignmentManager(1946): Server 10.22.16.34,59233,1476398451084 returned org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) for hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342., try=1 of 10 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:05,354 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:05,354 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:05,354 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 579 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:05,354 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398525354, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:05,355 INFO [AM.-pool11-t1] master.AssignmentManager(1946): Server 10.22.16.34,59233,1476398451084 returned org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) for hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342., try=2 of 10 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:05,355 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:05,355 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:05,355 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 580 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:05,356 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398525355, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:05,356 INFO [AM.-pool11-t1] master.AssignmentManager(1946): Server 10.22.16.34,59233,1476398451084 returned org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) for hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342., try=3 of 10 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:05,356 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:05,356 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:05,356 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 581 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:05,357 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398525356, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:05,357 INFO [AM.-pool11-t1] master.AssignmentManager(1946): Server 10.22.16.34,59233,1476398451084 returned org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) for hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342., try=4 of 10 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:05,357 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:05,357 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:05,357 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 582 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:05,357 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398525357, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:05,358 INFO [AM.-pool11-t1] master.AssignmentManager(1946): Server 10.22.16.34,59233,1476398451084 returned org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) for hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342., try=5 of 10 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:05,358 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:05,358 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:05,358 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 583 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:05,358 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398525358, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:05,359 INFO [AM.-pool11-t1] master.AssignmentManager(1946): Server 10.22.16.34,59233,1476398451084 returned org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) for hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342., try=6 of 10 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:05,360 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:05,360 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:05,360 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 584 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:05,360 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398525359, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:05,361 INFO [AM.-pool11-t1] master.AssignmentManager(1946): Server 10.22.16.34,59233,1476398451084 returned org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) for hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342., try=7 of 10 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:05,361 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:05,361 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:05,361 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 585 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:05,361 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398525361, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:05,362 INFO [AM.-pool11-t1] master.AssignmentManager(1946): Server 10.22.16.34,59233,1476398451084 returned org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) for hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342., try=8 of 10 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:05,362 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:05,362 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:05,362 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 586 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:05,362 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398525362, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:05,363 INFO [AM.-pool11-t1] master.AssignmentManager(1946): Server 10.22.16.34,59233,1476398451084 returned org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) for hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342., try=9 of 10 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:05,363 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.RSRpcServices(1255): Close 7ab1d0b3b7be8b9a7ce9af61c29d5342, moving to null 2016-10-13 15:42:05,363 INFO [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] regionserver.HRegionServer(2869): Received CLOSE for the region: 7ab1d0b3b7be8b9a7ce9af61c29d5342, which we are already trying to CLOSE, but not completed yet 2016-10-13 15:42:05,363 DEBUG [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233] ipc.CallRunner(126): RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233: callId: 587 service: AdminService methodName: CloseRegion size: 109 connection: 10.22.16.34:59267 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2016-10-13 15:42:05,363 DEBUG [AM.-pool11-t1] master.AssignmentManager(1912): update {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398525363, server=10.22.16.34,59233,1476398451084} the timestamp. 2016-10-13 15:42:05,363 INFO [AM.-pool11-t1] master.AssignmentManager(1946): Server 10.22.16.34,59233,1476398451084 returned org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) for hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342., try=10 of 10 org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1815) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:835) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1864) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1988) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1594) at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:48) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region 7ab1d0b3b7be8b9a7ce9af61c29d5342 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2877) at org.apache.hadoop.hbase.regionserver.RSRpcServices.closeRegion(RSRpcServices.java:1259) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22741) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:23647) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1812) ... 9 more 2016-10-13 15:42:05,364 WARN [AM.-pool11-t1] master.RegionStates(1109): Failed to open/close 7ab1d0b3b7be8b9a7ce9af61c29d5342 on 10.22.16.34,59233,1476398451084, set to FAILED_CLOSE 2016-10-13 15:42:05,364 INFO [AM.-pool11-t1] master.RegionStates(1118): Transition {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=PENDING_OPEN, ts=1476398525363, server=10.22.16.34,59233,1476398451084} to {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=FAILED_CLOSE, ts=1476398525364, server=10.22.16.34,59233,1476398451084} 2016-10-13 15:42:05,364 INFO [AM.-pool11-t1] master.AssignmentManager(1993): Skip assigning {ENCODED => 7ab1d0b3b7be8b9a7ce9af61c29d5342, NAME => 'hbase:namespace,,1476398463993.7ab1d0b3b7be8b9a7ce9af61c29d5342.', STARTKEY => '', ENDKEY => ''}, we couldn't close it: {7ab1d0b3b7be8b9a7ce9af61c29d5342 state=FAILED_CLOSE, ts=1476398525364, server=10.22.16.34,59233,1476398451084} 2016-10-13 15:43:03,989 DEBUG [region-location-0] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/data/hbase/meta/1588230740/info 2016-10-13 15:43:05,458 DEBUG [RpcServer.reader=2,bindAddress=10.22.16.34,port=59233] ipc.RpcServer$Listener(930): RpcServer.listener,port=59233: DISCONNECTING client 10.22.16.34:59269 because read count=-1. Number of active connections: 2 2016-10-13 15:44:03,985 DEBUG [region-location-2] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59227/user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/data/hbase/meta/1588230740/info 2016-10-13 15:44:05,666 DEBUG [RpcServer.reader=1,bindAddress=10.22.16.34,port=59233] ipc.RpcServer$Listener(930): RpcServer.listener,port=59233: DISCONNECTING client 10.22.16.34:59267 because read count=-1. Number of active connections: 1 Process Thread Dump: Thread dump because: Master not initialized after 200000ms seconds 219 active threads Thread 632 (region-location-4): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3dc08d37 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 627 (region-location-3): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3dc08d37 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 625 (RS_COMPACTED_FILES_DISCHARGER-10.22.16.34:59233-0): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4682eba5 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 621 (region-location-2): State: WAITING Blocked count: 1 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3dc08d37 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 617 (Thread-300): State: TIMED_WAITING Blocked count: 0 Waited count: 170 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:503) Thread 615 (Timer for 'DataNode' metrics system): State: TIMED_WAITING Blocked count: 0 Waited count: 21 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 611 (region-location-1): State: WAITING Blocked count: 1 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3dc08d37 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 599 (sync.4): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@df54a6e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Thread 598 (sync.3): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@fab14f5 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Thread 597 (sync.2): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@cd8c01b Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Thread 596 (sync.1): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35c85d82 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Thread 595 (sync.0): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1249d76f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Thread 600 (RS:0;10.22.16.34:59253.append-pool19-t1): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4592756f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45) com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55) com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 593 (sync.4): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@f80bdde Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Thread 592 (sync.3): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7853c5e1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Thread 591 (sync.2): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@247d972b Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Thread 590 (sync.1): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7292b7c6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Thread 589 (sync.0): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78769a07 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Thread 594 (RS:0;10.22.16.34:59233.append-pool18-t1): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1eec3425 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45) com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55) com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 588 (ResponseProcessor for block BP-1435415588-10.22.16.34-1476398450572:blk_1073741832_1008): State: RUNNABLE Blocked count: 1 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) java.io.FilterInputStream.read(FilterInputStream.java:83) java.io.FilterInputStream.read(FilterInputStream.java:83) org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1998) org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:798) Thread 587 (PacketResponder: BP-1435415588-10.22.16.34-1476398450572:blk_1073741832_1008, type=LAST_IN_PIPELINE, downstreams=0:[]): State: WAITING Blocked count: 7 Waited count: 8 Waiting on java.util.LinkedList@56603c1c Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1062) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1132) java.lang.Thread.run(Thread.java:745) Thread 586 (ResponseProcessor for block BP-1435415588-10.22.16.34-1476398450572:blk_1073741831_1007): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) java.io.FilterInputStream.read(FilterInputStream.java:83) java.io.FilterInputStream.read(FilterInputStream.java:83) org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1998) org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:798) Thread 585 (PacketResponder: BP-1435415588-10.22.16.34-1476398450572:blk_1073741831_1007, type=LAST_IN_PIPELINE, downstreams=0:[]): State: WAITING Blocked count: 7 Waited count: 8 Waiting on java.util.LinkedList@31ab9d91 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1062) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1132) java.lang.Thread.run(Thread.java:745) Thread 584 (DataXceiver for client DFSClient_NONMAPREDUCE_146676060_1 at /127.0.0.1:59272 [Receiving block BP-1435415588-10.22.16.34-1476398450572:blk_1073741832_1008]): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:453) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741) Thread 583 (DataXceiver for client DFSClient_NONMAPREDUCE_1090010022_1 at /127.0.0.1:59271 [Receiving block BP-1435415588-10.22.16.34-1476398450572:blk_1073741831_1007]): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:453) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741) Thread 582 (LeaseRenewer:tyu.hfs.1@localhost:59227): State: TIMED_WAITING Blocked count: 6 Waited count: 209 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Thread 581 (DataStreamer for file /user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/WALs/10.22.16.34,59253,1476398461576/10.22.16.34%2C59253%2C1476398461576.1476398464117 block BP-1435415588-10.22.16.34-1476398450572:blk_1073741832_1008): State: TIMED_WAITING Blocked count: 4 Waited count: 13 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:503) Thread 580 (DataStreamer for file /user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/WALs/10.22.16.34,59233,1476398451084/10.22.16.34%2C59233%2C1476398451084.1476398464117 block BP-1435415588-10.22.16.34-1476398450572:blk_1073741831_1007): State: TIMED_WAITING Blocked count: 3 Waited count: 11 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:503) Thread 579 (IPC Parameter Sending Thread #1): State: TIMED_WAITING Blocked count: 0 Waited count: 135 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 559 (sync.4): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@49c67ca9 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Thread 558 (sync.3): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55a55b40 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Thread 557 (sync.2): State: WAITING Blocked count: 2 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2498f097 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Thread 556 (sync.1): State: WAITING Blocked count: 2 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6af1669f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Thread 555 (sync.0): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@10287c5c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Thread 560 (RS_OPEN_META-10.22.16.34:59233-0.append-pool15-t1): State: WAITING Blocked count: 0 Waited count: 4 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6fa84a42 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45) com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55) com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 554 (ResponseProcessor for block BP-1435415588-10.22.16.34-1476398450572:blk_1073741829_1005): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) java.io.FilterInputStream.read(FilterInputStream.java:83) java.io.FilterInputStream.read(FilterInputStream.java:83) org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1998) org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:798) Thread 553 (PacketResponder: BP-1435415588-10.22.16.34-1476398450572:blk_1073741829_1005, type=LAST_IN_PIPELINE, downstreams=0:[]): State: WAITING Blocked count: 9 Waited count: 10 Waiting on java.util.LinkedList@c571368 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1062) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1132) java.lang.Thread.run(Thread.java:745) Thread 552 (DataXceiver for client DFSClient_NONMAPREDUCE_1090010022_1 at /127.0.0.1:59268 [Receiving block BP-1435415588-10.22.16.34-1476398450572:blk_1073741829_1005]): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:453) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741) Thread 551 (DataStreamer for file /user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/WALs/10.22.16.34,59233,1476398451084/10.22.16.34%2C59233%2C1476398451084.meta.1476398463953.meta block BP-1435415588-10.22.16.34-1476398450572:blk_1073741829_1005): State: TIMED_WAITING Blocked count: 7 Waited count: 15 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:503) Thread 550 (RS_OPEN_META-10.22.16.34:59233-0-MetaLogRoller): State: TIMED_WAITING Blocked count: 0 Waited count: 198 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:126) java.lang.Thread.run(Thread.java:745) Thread 548 (RS_OPEN_META-10.22.16.34:59233-0): State: WAITING Blocked count: 34 Waited count: 53 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13bdb558 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 546 (region-location-0): State: WAITING Blocked count: 5 Waited count: 13 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3dc08d37 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 544 (SplitLogWorker-10.22.16.34:59253): State: TIMED_WAITING Blocked count: 3 Waited count: 42 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.taskLoop(ZkSplitLogWorkerCoordination.java:445) org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:142) java.lang.Thread.run(Thread.java:745) Thread 543 (SplitLogWorker-10.22.16.34:59233): State: TIMED_WAITING Blocked count: 2 Waited count: 42 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.taskLoop(ZkSplitLogWorkerCoordination.java:445) org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:142) java.lang.Thread.run(Thread.java:745) Thread 542 (10.22.16.34:59233Replication Statistics #0): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 540 (10.22.16.34:59253Replication Statistics #0): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 512 (regionserver//10.22.16.34:0.leaseChecker): State: TIMED_WAITING Blocked count: 0 Waited count: 199 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:94) java.lang.Thread.run(Thread.java:745) Thread 509 (regionserver//10.22.16.34:0.leaseChecker): State: TIMED_WAITING Blocked count: 0 Waited count: 199 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:94) java.lang.Thread.run(Thread.java:745) Thread 535 (MemStoreFlusher.1): State: TIMED_WAITING Blocked count: 0 Waited count: 240 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) java.util.concurrent.DelayQueue.poll(DelayQueue.java:70) org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:239) java.lang.Thread.run(Thread.java:745) Thread 536 (MemStoreFlusher.1): State: TIMED_WAITING Blocked count: 0 Waited count: 232 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) java.util.concurrent.DelayQueue.poll(DelayQueue.java:70) org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:239) java.lang.Thread.run(Thread.java:745) Thread 533 (MemStoreFlusher.0): State: TIMED_WAITING Blocked count: 0 Waited count: 245 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) java.util.concurrent.DelayQueue.poll(DelayQueue.java:70) org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:239) java.lang.Thread.run(Thread.java:745) Thread 531 (MemStoreFlusher.0): State: TIMED_WAITING Blocked count: 0 Waited count: 238 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) java.util.concurrent.DelayQueue.poll(DelayQueue.java:70) org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:239) java.lang.Thread.run(Thread.java:745) Thread 417 (regionserver//10.22.16.34:0.logRoller): State: TIMED_WAITING Blocked count: 0 Waited count: 199 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:126) java.lang.Thread.run(Thread.java:745) Thread 489 (regionserver//10.22.16.34:0.logRoller): State: TIMED_WAITING Blocked count: 0 Waited count: 199 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:126) java.lang.Thread.run(Thread.java:745) Thread 529 (snapshot-hfile-cleaner-cache-refresher): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 528 (10.22.16.34:59249.activeMasterManager-EventThread): State: WAITING Blocked count: 0 Waited count: 3 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56e70b77 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Thread 527 (10.22.16.34:59249.activeMasterManager-SendThread(localhost:55969)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Thread 525 (ProcedureExecutorTimeout): State: TIMED_WAITING Blocked count: 0 Waited count: 9 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) org.apache.hadoop.hbase.procedure2.util.TimeoutBlockingQueue.poll(TimeoutBlockingQueue.java:106) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.timeoutLoop(ProcedureExecutor.java:886) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$500(ProcedureExecutor.java:76) org.apache.hadoop.hbase.procedure2.ProcedureExecutor$3.run(ProcedureExecutor.java:505) Thread 524 (ProcedureExecutor-7): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25895cb5 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:149) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:139) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:805) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76) org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:496) Thread 523 (ProcedureExecutor-6): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25895cb5 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:149) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:139) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:805) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76) org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:496) Thread 522 (ProcedureExecutor-5): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25895cb5 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:149) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:139) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:805) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76) org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:496) Thread 521 (ProcedureExecutor-4): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25895cb5 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:149) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:139) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:805) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76) org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:496) Thread 520 (ProcedureExecutor-3): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25895cb5 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:149) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:139) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:805) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76) org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:496) Thread 519 (ProcedureExecutor-2): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25895cb5 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:149) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:139) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:805) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76) org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:496) Thread 518 (ProcedureExecutor-1): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25895cb5 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:149) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:139) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:805) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76) org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:496) Thread 517 (ProcedureExecutor-0): State: WAITING Blocked count: 35 Waited count: 58 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25895cb5 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:149) org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler.poll(MasterProcedureScheduler.java:139) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:805) org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:76) org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:496) Thread 516 (IPC Client (1711693104) connection to /10.22.16.34:59249 from tyu.hfs.1): State: TIMED_WAITING Blocked count: 200 Waited count: 397 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.waitForWork(RpcClientImpl.java:550) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:565) Thread 514 (IPC Client (1711693104) connection to /10.22.16.34:59249 from tyu): State: TIMED_WAITING Blocked count: 200 Waited count: 397 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.waitForWork(RpcClientImpl.java:550) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:565) Thread 515 (WALProcedureStoreSyncThread): State: TIMED_WAITING Blocked count: 25 Waited count: 47 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.syncLoop(WALProcedureStore.java:578) org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.access$000(WALProcedureStore.java:67) org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore$1.run(WALProcedureStore.java:215) Thread 510 (RS:0;10.22.16.34:59253-longCompactions-1476398462037): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13ed6be Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:90) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 508 (RS:0;10.22.16.34:59233-longCompactions-1476398462037): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@76e7c2af Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:90) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 507 (10.22.16.34:59249.activeMasterManager-EventThread): State: WAITING Blocked count: 0 Waited count: 3 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@79655beb Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Thread 506 (10.22.16.34:59249.activeMasterManager-SendThread(localhost:55969)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Thread 505 (10.22.16.34,59249,1476398461537_splitLogManager__ChoreService_1): State: TIMED_WAITING Blocked count: 0 Waited count: 200 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 498 (JvmPauseMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 398 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159) java.lang.Thread.run(Thread.java:745) Thread 497 (JvmPauseMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 398 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159) java.lang.Thread.run(Thread.java:745) Thread 495 (M:0;10.22.16.34:59249-EventThread): State: WAITING Blocked count: 1 Waited count: 3 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@724e1c9f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Thread 494 (M:0;10.22.16.34:59249-SendThread(localhost:55969)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Thread 493 (RS:0;10.22.16.34:59253-EventThread): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a744ac3 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Thread 492 (RS:0;10.22.16.34:59253-SendThread(localhost:55969)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Thread 491 (RS:0;10.22.16.34:59253): State: TIMED_WAITING Blocked count: 299 Waited count: 455 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:56) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:991) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:157) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:110) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:141) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:334) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:139) java.lang.Thread.run(Thread.java:745) Thread 455 (M:0;10.22.16.34:59249): State: TIMED_WAITING Blocked count: 8 Waited count: 209 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:56) org.apache.hadoop.hbase.master.HMaster.waitForMasterActive(HMaster.java:519) org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:815) org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:768) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:914) java.lang.Thread.run(Thread.java:745) Thread 490 (10.22.16.34,59253,1476398461576_ChoreService_1): State: TIMED_WAITING Blocked count: 1 Waited count: 312 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 488 (RpcServer.FifoWFPBQ.replication.handler=2,queue=0,port=59253): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@6320d1e1 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 487 (RpcServer.FifoWFPBQ.replication.handler=1,queue=0,port=59253): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@65982f3c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 486 (RpcServer.FifoWFPBQ.replication.handler=0,queue=0,port=59253): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@bbee44e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 485 (RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59253): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@25fab98c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 484 (RpcServer.FifoWFPBQ.priority.handler=4,queue=0,port=59253): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@40b0db0b Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 483 (RpcServer.FifoWFPBQ.priority.handler=3,queue=1,port=59253): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@64f5fdee Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 482 (RpcServer.FifoWFPBQ.priority.handler=2,queue=0,port=59253): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@1895e9aa Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 481 (RpcServer.FifoWFPBQ.priority.handler=1,queue=1,port=59253): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@68d305ed Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 480 (RpcServer.FifoWFPBQ.priority.handler=0,queue=0,port=59253): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@2e7de37f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 479 (RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59253): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@652c61d3 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 478 (RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=59253): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@52431c65 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 476 (RpcServer.FifoWFPBQ.default.handler=2,queue=0,port=59253): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@224f87ad Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 474 (RpcServer.FifoWFPBQ.default.handler=1,queue=0,port=59253): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@6368ee80 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 473 (RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=59253): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@65cc29c7 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 464 (RpcServer.listener,port=59253): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:791) Thread 468 (RpcServer.responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRunLoop(RpcServer.java:1031) org.apache.hadoop.hbase.ipc.RpcServer$Responder.run(RpcServer.java:978) Thread 470 (main-EventThread): State: WAITING Blocked count: 2 Waited count: 5 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2f26b12f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Thread 469 (main-SendThread(localhost:55969)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Thread 467 (RpcServer.reader=2,bindAddress=10.22.16.34,port=59253): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:685) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:672) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 466 (RpcServer.reader=1,bindAddress=10.22.16.34,port=59253): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:685) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:672) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 465 (RpcServer.reader=0,bindAddress=10.22.16.34,port=59253): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:685) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:672) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 457 (Thread-240): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:231) java.lang.Thread.run(Thread.java:745) Thread 454 (10.22.16.34:59249.activeMasterManager): State: TIMED_WAITING Blocked count: 285 Waited count: 2275 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:101) org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1014) org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:806) org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:195) org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1827) java.lang.Thread.run(Thread.java:745) Thread 453 (10.22.16.34,59249,1476398461537_ChoreService_1): State: TIMED_WAITING Blocked count: 17 Waited count: 358 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 451 (RpcServer.FifoWFPBQ.replication.handler=2,queue=0,port=59249): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@5532c5d6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 450 (RpcServer.FifoWFPBQ.replication.handler=1,queue=0,port=59249): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@51b45ca6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 449 (RpcServer.FifoWFPBQ.replication.handler=0,queue=0,port=59249): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@4b013076 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 448 (RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59249): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@6b679efe Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 447 (RpcServer.FifoWFPBQ.priority.handler=4,queue=0,port=59249): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@28b8ea31 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 446 (RpcServer.FifoWFPBQ.priority.handler=3,queue=1,port=59249): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@5b1ad2ec Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 445 (RpcServer.FifoWFPBQ.priority.handler=2,queue=0,port=59249): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@19dacb79 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 444 (RpcServer.FifoWFPBQ.priority.handler=1,queue=1,port=59249): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@4e9d26d2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 443 (RpcServer.FifoWFPBQ.priority.handler=0,queue=0,port=59249): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@219ba452 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 442 (RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59249): State: WAITING Blocked count: 5 Waited count: 248 Waiting on java.util.concurrent.Semaphore$NonfairSync@2c1b8f3f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 441 (RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=59249): State: WAITING Blocked count: 3 Waited count: 153 Waiting on java.util.concurrent.Semaphore$NonfairSync@63d518c6 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 440 (RpcServer.FifoWFPBQ.default.handler=2,queue=0,port=59249): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@44d919d8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 439 (RpcServer.FifoWFPBQ.default.handler=1,queue=0,port=59249): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@381c9c25 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 438 (RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=59249): State: WAITING Blocked count: 0 Waited count: 1 Waiting on java.util.concurrent.Semaphore$NonfairSync@480e8e6e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 431 (RpcServer.listener,port=59249): State: RUNNABLE Blocked count: 2 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:791) Thread 435 (RpcServer.responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRunLoop(RpcServer.java:1031) org.apache.hadoop.hbase.ipc.RpcServer$Responder.run(RpcServer.java:978) Thread 437 (main-EventThread): State: WAITING Blocked count: 14 Waited count: 19 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@356bc81 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Thread 436 (main-SendThread(localhost:55969)): State: RUNNABLE Blocked count: 2 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Thread 434 (RpcServer.reader=2,bindAddress=10.22.16.34,port=59249): State: RUNNABLE Blocked count: 2 Waited count: 1 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:685) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:672) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 433 (RpcServer.reader=1,bindAddress=10.22.16.34,port=59249): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:685) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:672) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 432 (RpcServer.reader=0,bindAddress=10.22.16.34,port=59249): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:685) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:672) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 426 (LeaseRenewer:tyu@localhost:59227): State: TIMED_WAITING Blocked count: 6 Waited count: 213 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Thread 422 (JvmPauseMonitor): State: TIMED_WAITING Blocked count: 0 Waited count: 419 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159) java.lang.Thread.run(Thread.java:745) Thread 421 (RS:0;10.22.16.34:59233-EventThread): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@15b4fa97 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Thread 420 (RS:0;10.22.16.34:59233-SendThread(localhost:55969)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Thread 419 (RS:0;10.22.16.34:59233): State: TIMED_WAITING Blocked count: 287 Waited count: 556 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:56) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:991) java.lang.Thread.run(Thread.java:745) Thread 418 (10.22.16.34,59233,1476398451084_ChoreService_1): State: TIMED_WAITING Blocked count: 0 Waited count: 316 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 416 (RpcServer.FifoWFPBQ.replication.handler=2,queue=0,port=59233): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@5150541 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 415 (RpcServer.FifoWFPBQ.replication.handler=1,queue=0,port=59233): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@37c72736 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 414 (RpcServer.FifoWFPBQ.replication.handler=0,queue=0,port=59233): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@1806b2cd Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 413 (RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233): State: WAITING Blocked count: 4 Waited count: 610 Waiting on java.util.concurrent.Semaphore$NonfairSync@329a9b7d Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 412 (RpcServer.FifoWFPBQ.priority.handler=4,queue=0,port=59233): State: WAITING Blocked count: 0 Waited count: 4 Waiting on java.util.concurrent.Semaphore$NonfairSync@1f08bf62 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 411 (RpcServer.FifoWFPBQ.priority.handler=3,queue=1,port=59233): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@1178d522 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 410 (RpcServer.FifoWFPBQ.priority.handler=2,queue=0,port=59233): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@3c458d1a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 409 (RpcServer.FifoWFPBQ.priority.handler=1,queue=1,port=59233): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@5a62b89c Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 408 (RpcServer.FifoWFPBQ.priority.handler=0,queue=0,port=59233): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@cbef1d Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 407 (RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59233): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@574dfbe3 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 406 (RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=59233): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@4770814d Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 405 (RpcServer.FifoWFPBQ.default.handler=2,queue=0,port=59233): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@8997767 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 404 (RpcServer.FifoWFPBQ.default.handler=1,queue=0,port=59233): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@5775c26e Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 403 (RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=59233): State: WAITING Blocked count: 0 Waited count: 2 Waiting on java.util.concurrent.Semaphore$NonfairSync@6fdfb980 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Thread 396 (RpcServer.listener,port=59233): State: RUNNABLE Blocked count: 2 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:791) Thread 400 (RpcServer.responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRunLoop(RpcServer.java:1031) org.apache.hadoop.hbase.ipc.RpcServer$Responder.run(RpcServer.java:978) Thread 402 (main-EventThread): State: WAITING Blocked count: 3 Waited count: 6 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2c6627df Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Thread 401 (main-SendThread(localhost:55969)): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Thread 399 (RpcServer.reader=2,bindAddress=10.22.16.34,port=59233): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:685) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:672) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 398 (RpcServer.reader=1,bindAddress=10.22.16.34,port=59233): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:685) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:672) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 397 (RpcServer.reader=0,bindAddress=10.22.16.34,port=59233): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:685) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:672) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 392 (refreshUsed-/Users/tyu/1.3-h/hbase-server/target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4/dfscluster_64f81ea3-ebf2-43ae-9724-b2d9ced5cbba/dfs/data/data2/current/BP-1435415588-10.22.16.34-1476398450572): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:115) java.lang.Thread.run(Thread.java:745) Thread 391 (refreshUsed-/Users/tyu/1.3-h/hbase-server/target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4/dfscluster_64f81ea3-ebf2-43ae-9724-b2d9ced5cbba/dfs/data/data1/current/BP-1435415588-10.22.16.34-1476398450572): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:115) java.lang.Thread.run(Thread.java:745) Thread 386 (java.util.concurrent.ThreadPoolExecutor$Worker@e94d4df[State = -1, empty queue]): State: TIMED_WAITING Blocked count: 0 Waited count: 2 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 383 (Thread-216): State: TIMED_WAITING Blocked count: 0 Waited count: 43 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:78) java.lang.Thread.run(Thread.java:745) Thread 374 (IPC Client (1711693104) connection to localhost/127.0.0.1:59227 from tyu): State: TIMED_WAITING Blocked count: 90 Waited count: 91 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:904) org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) Thread 379 (IPC Server handler 9 on 59230): State: TIMED_WAITING Blocked count: 0 Waited count: 390 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 378 (IPC Server handler 8 on 59230): State: TIMED_WAITING Blocked count: 0 Waited count: 369 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 377 (IPC Server handler 7 on 59230): State: TIMED_WAITING Blocked count: 0 Waited count: 388 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 376 (IPC Server handler 6 on 59230): State: TIMED_WAITING Blocked count: 0 Waited count: 397 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 375 (IPC Server handler 5 on 59230): State: TIMED_WAITING Blocked count: 0 Waited count: 384 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 373 (IPC Server handler 4 on 59230): State: TIMED_WAITING Blocked count: 0 Waited count: 388 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 372 (IPC Server handler 3 on 59230): State: TIMED_WAITING Blocked count: 0 Waited count: 396 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 371 (IPC Server handler 2 on 59230): State: TIMED_WAITING Blocked count: 0 Waited count: 403 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 370 (IPC Server handler 1 on 59230): State: TIMED_WAITING Blocked count: 0 Waited count: 364 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 369 (IPC Server handler 0 on 59230): State: TIMED_WAITING Blocked count: 0 Waited count: 390 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 364 (IPC Server listener on 59230): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:668) Thread 367 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:835) org.apache.hadoop.ipc.Server$Responder.run(Server.java:818) Thread 359 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@6a9950f1): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:100) org.apache.hadoop.hdfs.net.TcpPeerServer.accept(TcpPeerServer.java:134) org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:137) java.lang.Thread.run(Thread.java:745) Thread 368 (DataNode: [[[DISK]file:/Users/tyu/1.3-h/hbase-server/target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4/dfscluster_64f81ea3-ebf2-43ae-9724-b2d9ced5cbba/dfs/data/data1/, [DISK]file:/Users/tyu/1.3-h/hbase-server/target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4/dfscluster_64f81ea3-ebf2-43ae-9724-b2d9ced5cbba/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:59227): State: TIMED_WAITING Blocked count: 113 Waited count: 275 Stack: java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:720) org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:834) java.lang.Thread.run(Thread.java:745) Thread 366 (IPC Server idle connection scanner for port 59230): State: TIMED_WAITING Blocked count: 1 Waited count: 26 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 365 (Socket Reader #1 for port 59230): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:616) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:595) Thread 363 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@76c52298): State: TIMED_WAITING Blocked count: 0 Waited count: 420 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:180) java.lang.Thread.run(Thread.java:745) Thread 362 (Timer-3): State: TIMED_WAITING Blocked count: 0 Waited count: 9 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 361 (745465688@qtp-1929476736-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59229): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 360 (1723518396@qtp-1929476736-0): State: TIMED_WAITING Blocked count: 0 Waited count: 4 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 354 (CacheReplicationMonitor(1140143724)): State: TIMED_WAITING Blocked count: 0 Waited count: 8 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor.run(CacheReplicationMonitor.java:181) Thread 353 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller@5ce4369b): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeEditLogRoller.run(FSNamesystem.java:4726) java.lang.Thread.run(Thread.java:745) Thread 352 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@78830d9a): State: TIMED_WAITING Blocked count: 0 Waited count: 43 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor.run(FSNamesystem.java:4687) java.lang.Thread.run(Thread.java:745) Thread 351 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@5da7cee2): State: TIMED_WAITING Blocked count: 0 Waited count: 108 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:392) java.lang.Thread.run(Thread.java:745) Thread 350 (IPC Server handler 9 on 59227): State: TIMED_WAITING Blocked count: 0 Waited count: 329 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 349 (IPC Server handler 8 on 59227): State: TIMED_WAITING Blocked count: 1 Waited count: 339 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 348 (IPC Server handler 7 on 59227): State: TIMED_WAITING Blocked count: 0 Waited count: 350 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 347 (IPC Server handler 6 on 59227): State: TIMED_WAITING Blocked count: 0 Waited count: 304 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 346 (IPC Server handler 5 on 59227): State: TIMED_WAITING Blocked count: 0 Waited count: 355 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 345 (IPC Server handler 4 on 59227): State: TIMED_WAITING Blocked count: 0 Waited count: 355 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 344 (IPC Server handler 3 on 59227): State: TIMED_WAITING Blocked count: 0 Waited count: 308 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 343 (IPC Server handler 2 on 59227): State: TIMED_WAITING Blocked count: 0 Waited count: 269 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 342 (IPC Server handler 1 on 59227): State: TIMED_WAITING Blocked count: 0 Waited count: 375 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 341 (IPC Server handler 0 on 59227): State: TIMED_WAITING Blocked count: 0 Waited count: 324 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:109) org.apache.hadoop.ipc.Server$Handler.run(Server.java:1985) Thread 331 (IPC Server listener on 59227): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener.run(Server.java:668) Thread 334 (IPC Server Responder): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.ipc.Server$Responder.doRunLoop(Server.java:835) org.apache.hadoop.ipc.Server$Responder.run(Server.java:818) Thread 329 (org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@2ba33e2c): State: TIMED_WAITING Blocked count: 0 Waited count: 77 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3397) java.lang.Thread.run(Thread.java:745) Thread 330 (org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@1f193686): State: TIMED_WAITING Blocked count: 0 Waited count: 43 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor.run(HeartbeatManager.java:319) java.lang.Thread.run(Thread.java:745) Thread 340 (org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor@3fae596): State: TIMED_WAITING Blocked count: 0 Waited count: 73 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor.run(DecommissionManager.java:76) java.lang.Thread.run(Thread.java:745) Thread 339 (org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor@105b693d): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.server.blockmanagement.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:221) java.lang.Thread.run(Thread.java:745) Thread 335 (org.apache.hadoop.util.JvmPauseMonitor$Monitor@5bca7664): State: TIMED_WAITING Blocked count: 0 Waited count: 420 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:180) java.lang.Thread.run(Thread.java:745) Thread 333 (IPC Server idle connection scanner for port 59227): State: TIMED_WAITING Blocked count: 1 Waited count: 24 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 332 (Socket Reader #1 for port 59227): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:616) org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:595) Thread 328 (Timer-2): State: TIMED_WAITING Blocked count: 0 Waited count: 9 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 327 (1661803311@qtp-573023937-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59226): State: RUNNABLE Blocked count: 1 Waited count: 1 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thread 326 (1625428489@qtp-573023937-0): State: TIMED_WAITING Blocked count: 0 Waited count: 4 Stack: java.lang.Object.wait(Native Method) org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626) Thread 320 (ProcessThread(sid:0 cport:-1):): State: WAITING Blocked count: 1 Waited count: 440 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d12b024 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120) Thread 319 (SyncThread:0): State: WAITING Blocked count: 7 Waited count: 414 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72fe8a4f Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:127) Thread 318 (SessionTracker): State: TIMED_WAITING Blocked count: 0 Waited count: 106 Stack: java.lang.Object.wait(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) Thread 317 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:55969): State: RUNNABLE Blocked count: 7 Waited count: 0 Stack: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178) java.lang.Thread.run(Thread.java:745) Thread 216 (HBase-Metrics2-1): State: TIMED_WAITING Blocked count: 0 Waited count: 163 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Thread 214 (snapshot-hfile-cleaner-cache-refresher): State: TIMED_WAITING Blocked count: 0 Waited count: 1 Stack: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Thread 213 (10.22.16.34:59186.activeMasterManager-EventThread): State: WAITING Blocked count: 0 Waited count: 3 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@272cc4ed Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Thread 212 (10.22.16.34:59186.activeMasterManager-SendThread(localhost:51820)): State: TIMED_WAITING Blocked count: 0 Waited count: 390 Stack: java.lang.Thread.sleep(Native Method) org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) Thread 167 (org.apache.hadoop.hdfs.PeerCache@59937acd): State: TIMED_WAITING Blocked count: 0 Waited count: 74 Stack: java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:244) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119) java.lang.Thread.run(Thread.java:745) Thread 4 (Signal Dispatcher): State: RUNNABLE Blocked count: 0 Waited count: 0 Stack: Thread 3 (Finalizer): State: WAITING Blocked count: 579 Waited count: 14 Waiting on java.lang.ref.ReferenceQueue$Lock@34129c78 Stack: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread 2 (Reference Handler): State: WAITING Blocked count: 21 Waited count: 11 Waiting on java.lang.ref.Reference$Lock@7d61eb55 Stack: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread 1 (main): State: RUNNABLE Blocked count: 145 Waited count: 684 Stack: sun.management.ThreadImpl.getThreadInfo1(Native Method) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:178) sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139) org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:165) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:301) org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:223) org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:409) org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:227) org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:96) org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1071) org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1037) org.apache.hadoop.hbase.regionserver.TestClusterId.testClusterId(TestClusterId.java:84) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) 2016-10-13 15:44:21,713 ERROR [main] hbase.MiniHBaseCluster(232): Error starting cluster java.lang.RuntimeException: Master not initialized after 200000ms seconds at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:225) at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:409) at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:227) at org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:96) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1071) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1037) at org.apache.hadoop.hbase.regionserver.TestClusterId.testClusterId(TestClusterId.java:84) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) 2016-10-13 15:44:21,714 DEBUG [main] util.JVMClusterUtil(241): Shutting down HBase Cluster 2016-10-13 15:44:21,714 INFO [main] regionserver.HRegionServer(1908): STOPPED: Cluster shutdown requested 2016-10-13 15:44:21,714 INFO [M:0;10.22.16.34:59249] procedure2.ProcedureExecutor(534): Stopping the procedure executor 2016-10-13 15:44:21,714 INFO [M:0;10.22.16.34:59249] wal.WALProcedureStore(233): Stopping the WAL Procedure Store 2016-10-13 15:44:21,717 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2016-10-13 15:44:21,717 INFO [main] regionserver.HRegionServer(1908): STOPPED: Shutdown requested 2016-10-13 15:44:21,717 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59253-0x157c034877e0003, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2016-10-13 15:44:21,717 DEBUG [main-EventThread] zookeeper.ZKUtil(371): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-10-13 15:44:21,717 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2016-10-13 15:44:21,717 INFO [RS:0;10.22.16.34:59253] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2016-10-13 15:44:21,717 DEBUG [main-EventThread] zookeeper.ZKUtil(371): regionserver:59253-0x157c034877e0003, quorum=localhost:55969, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-10-13 15:44:21,717 INFO [SplitLogWorker-10.22.16.34:59253] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2016-10-13 15:44:21,717 INFO [RS:0;10.22.16.34:59253] regionserver.HeapMemoryManager(202): Stoping HeapMemoryTuner chore. 2016-10-13 15:44:21,717 DEBUG [main-EventThread] zookeeper.ZKUtil(371): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-10-13 15:44:21,717 INFO [RS:0;10.22.16.34:59253] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2016-10-13 15:44:21,717 INFO [SplitLogWorker-10.22.16.34:59253] regionserver.SplitLogWorker(155): SplitLogWorker 10.22.16.34,59253,1476398461576 exiting 2016-10-13 15:44:21,718 INFO [RS:0;10.22.16.34:59253] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2016-10-13 15:44:21,718 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2016-10-13 15:44:21,717 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2016-10-13 15:44:21,718 INFO [RS:0;10.22.16.34:59253] regionserver.HRegionServer(1056): stopping server 10.22.16.34,59253,1476398461576 2016-10-13 15:44:21,718 DEBUG [RS:0;10.22.16.34:59253] zookeeper.MetaTableLocator(622): Stopping MetaTableLocator 2016-10-13 15:44:21,718 INFO [RS:0;10.22.16.34:59253] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x157c034877e0004 2016-10-13 15:44:21,719 DEBUG [RS:0;10.22.16.34:59253] ipc.RpcClientImpl(1160): Stopping rpc client 2016-10-13 15:44:21,719 INFO [RS:0;10.22.16.34:59253] regionserver.HRegionServer(1084): stopping server 10.22.16.34,59253,1476398461576; all regions closed. 2016-10-13 15:44:21,719 DEBUG [RS:0;10.22.16.34:59253] wal.FSHLog(1080): Closing WAL writer in /user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/WALs/10.22.16.34,59253,1476398461576 2016-10-13 15:44:21,723 INFO [IPC Server handler 5 on 59227] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59228 is added to blk_1073741832_1008{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-c3973975-1bcc-4f64-82f9-cc946695f6b0:NORMAL|RBW]]} size 83 2016-10-13 15:44:21,725 INFO [IPC Server handler 2 on 59227] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59228 is added to blk_1073741835_1011{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-db86b596-4d7b-4cf7-9974-02d746fadd02:NORMAL|RBW]]} size 30 2016-10-13 15:44:21,729 DEBUG [RS:0;10.22.16.34:59253] wal.FSHLog(1038): Moved 1 WAL file(s) to /user/tyu/test-data/3349b3a4-c93b-43df-a627-230b964563b1/oldWALs 2016-10-13 15:44:21,729 INFO [RS:0;10.22.16.34:59253] wal.FSHLog(1041): Closed WAL: FSHLog 10.22.16.34%2C59253%2C1476398461576:(num 1476398464117) 2016-10-13 15:44:21,729 DEBUG [RS:0;10.22.16.34:59253] ipc.RpcClientImpl(1160): Stopping rpc client 2016-10-13 15:44:21,729 DEBUG [RpcServer.reader=2,bindAddress=10.22.16.34,port=59249] ipc.RpcServer$Listener(930): RpcServer.listener,port=59249: DISCONNECTING client 10.22.16.34:59261 because read count=-1. Number of active connections: 2 2016-10-13 15:44:21,741 INFO [RS:0;10.22.16.34:59253] regionserver.Leases(146): RS:0;10.22.16.34:59253 closing leases 2016-10-13 15:44:21,742 INFO [RS:0;10.22.16.34:59253] regionserver.Leases(149): RS:0;10.22.16.34:59253 closed leases 2016-10-13 15:44:21,742 INFO [RS:0;10.22.16.34:59253] hbase.ChoreService(328): Chore service for: 10.22.16.34,59253,1476398461576 had [[ScheduledChore: Name: MovedRegionsCleaner for region 10.22.16.34,59253,1476398461576 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.22.16.34,59253,1476398461576-MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS]] on shutdown 2016-10-13 15:44:21,819 INFO [RS:0;10.22.16.34:59233] regionserver.HRegionServer(963): Closing user regions 2016-10-13 15:44:22,013 INFO [10.22.16.34,59249,1476398461537_splitLogManager__ChoreService_1] hbase.ScheduledChore(183): Chore: SplitLogManager Timeout Monitor was stopped 2016-10-13 15:44:22,013 INFO [10.22.16.34,59249,1476398461537_ChoreService_1] hbase.ScheduledChore(183): Chore: 10.22.16.34,59249,1476398461537-DoMetricsChore was stopped 2016-10-13 15:44:22,131 INFO [M:0;10.22.16.34:59249] regionserver.HRegionServer(1056): stopping server 10.22.16.34,59249,1476398461537 2016-10-13 15:44:22,131 DEBUG [M:0;10.22.16.34:59249] zookeeper.MetaTableLocator(622): Stopping MetaTableLocator 2016-10-13 15:44:22,131 INFO [M:0;10.22.16.34:59249] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x157c034877e0005 2016-10-13 15:44:22,134 DEBUG [M:0;10.22.16.34:59249] ipc.RpcClientImpl(1160): Stopping rpc client 2016-10-13 15:44:22,134 INFO [M:0;10.22.16.34:59249] regionserver.HRegionServer(1084): stopping server 10.22.16.34,59249,1476398461537; all regions closed. 2016-10-13 15:44:22,135 INFO [M:0;10.22.16.34:59249] hbase.ChoreService(328): Chore service for: 10.22.16.34,59249,1476398461537 had [[ScheduledChore: Name: CatalogJanitor-10.22.16.34:59249 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.22.16.34,59249,1476398461537-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.22.16.34,59249,1476398461537-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: 10.22.16.34,59249,1476398461537-RegionNormalizerChore Period: 1800000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS]] on shutdown 2016-10-13 15:44:22,135 INFO [M:0;10.22.16.34:59249] master.ServerManager(566): Waiting on regionserver(s) to go down 10.22.16.34,59233,1476398451084, 10.22.16.34,59253,1476398461576 2016-10-13 15:44:22,568 INFO [regionserver//10.22.16.34:0.logRoller] regionserver.LogRoller(176): LogRoller exiting. 2016-10-13 15:44:22,568 INFO [regionserver//10.22.16.34:0.leaseChecker] regionserver.Leases(146): regionserver//10.22.16.34:0.leaseChecker closing leases 2016-10-13 15:44:22,569 INFO [regionserver//10.22.16.34:0.leaseChecker] regionserver.Leases(149): regionserver//10.22.16.34:0.leaseChecker closed leases 2016-10-13 15:44:22,569 INFO [RS:0;10.22.16.34:59253] regionserver.CompactSplitThread(403): Waiting for Split Thread to finish... 2016-10-13 15:44:22,569 INFO [RS:0;10.22.16.34:59253] regionserver.CompactSplitThread(403): Waiting for Merge Thread to finish... 2016-10-13 15:44:22,569 INFO [RS:0;10.22.16.34:59253] regionserver.CompactSplitThread(403): Waiting for Large Compaction Thread to finish... 2016-10-13 15:44:22,569 INFO [RS:0;10.22.16.34:59253] regionserver.CompactSplitThread(403): Waiting for Small Compaction Thread to finish... 2016-10-13 15:44:22,573 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59253-0x157c034877e0003, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/10.22.16.34,59253,1476398461576 2016-10-13 15:44:22,574 INFO [RS:0;10.22.16.34:59253] ipc.RpcServer(2360): Stopping server on 59253 2016-10-13 15:44:22,574 INFO [RpcServer.listener,port=59253] ipc.RpcServer$Listener(834): RpcServer.listener,port=59253: stopping 2016-10-13 15:44:22,575 INFO [RpcServer.responder] ipc.RpcServer$Responder(1077): RpcServer.responder: stopped 2016-10-13 15:44:22,575 INFO [RpcServer.responder] ipc.RpcServer$Responder(980): RpcServer.responder: stopping 2016-10-13 15:44:22,576 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.22.16.34,59253,1476398461576 2016-10-13 15:44:22,576 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59253-0x157c034877e0003, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.22.16.34,59253,1476398461576 2016-10-13 15:44:22,576 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/10.22.16.34,59253,1476398461576 2016-10-13 15:44:22,577 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59253-0x157c034877e0003, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2016-10-13 15:44:22,577 INFO [main-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [10.22.16.34,59253,1476398461576] 2016-10-13 15:44:22,577 INFO [RS:0;10.22.16.34:59253] regionserver.HRegionServer(1128): stopping server 10.22.16.34,59253,1476398461576; zookeeper connection closed. 2016-10-13 15:44:22,578 INFO [RS:0;10.22.16.34:59253] regionserver.HRegionServer(1131): RS:0;10.22.16.34:59253 exiting 2016-10-13 15:44:22,578 INFO [main-EventThread] master.ServerManager(621): Cluster shutdown set; 10.22.16.34,59253,1476398461576 expired; onlineServers=1 2016-10-13 15:44:22,578 DEBUG [main-EventThread] zookeeper.ZKUtil(369): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/10.22.16.34,59233,1476398451084 2016-10-13 15:44:22,578 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@50724e6a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(193): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@50724e6a 2016-10-13 15:44:22,578 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(121): /hbase/rs/10.22.16.34,59253,1476398461576 znode expired, triggering replicatorRemoved event 2016-10-13 15:44:22,578 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2016-10-13 15:44:22,579 INFO [main] util.JVMClusterUtil(326): Shutdown of 1 master(s) and 1 regionserver(s) complete 2016-10-13 15:44:22,579 INFO [main] hbase.HBaseTestingUtility(1130): Shutting down minicluster 2016-10-13 15:44:22,583 INFO [main] zookeeper.MiniZooKeeperCluster(319): Shutdown MiniZK cluster with all ZK servers 2016-10-13 15:44:22,583 WARN [main] datanode.DirectoryScanner(375): DirectoryScanner: shutdown has been called 2016-10-13 15:44:22,584 WARN [ResponseProcessor for block BP-1435415588-10.22.16.34-1476398450572:blk_1073741831_1007] hdfs.DFSOutputStream$DataStreamer$ResponseProcessor(883): DFSOutputStream ResponseProcessor exception for block BP-1435415588-10.22.16.34-1476398450572:blk_1073741831_1007 java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2000) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:798) 2016-10-13 15:44:22,584 WARN [ResponseProcessor for block BP-1435415588-10.22.16.34-1476398450572:blk_1073741829_1005] hdfs.DFSOutputStream$DataStreamer$ResponseProcessor(883): DFSOutputStream ResponseProcessor exception for block BP-1435415588-10.22.16.34-1476398450572:blk_1073741829_1005 java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2000) at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:798) 2016-10-13 15:44:22,586 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2016-10-13 15:44:22,586 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2016-10-13 15:44:22,683 DEBUG [RS:0;10.22.16.34:59233-EventThread] zookeeper.ZooKeeperWatcher(600): hconnection-0x2a1858e8-0x157c034877e0001, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2016-10-13 15:44:22,684 DEBUG [RS:0;10.22.16.34:59233-EventThread] zookeeper.ZooKeeperWatcher(668): hconnection-0x2a1858e8-0x157c034877e0001, quorum=localhost:55969, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2016-10-13 15:44:22,685 DEBUG [10.22.16.34:59249.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(600): hconnection-0x3e930e36-0x157c034877e0006, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2016-10-13 15:44:22,685 DEBUG [10.22.16.34:59249.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(600): replicationLogCleaner-0x157c034877e0007, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2016-10-13 15:44:22,685 DEBUG [10.22.16.34:59249.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(668): replicationLogCleaner-0x157c034877e0007, quorum=localhost:55969, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2016-10-13 15:44:22,685 DEBUG [10.22.16.34:59249.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(668): hconnection-0x3e930e36-0x157c034877e0006, quorum=localhost:55969, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2016-10-13 15:44:22,686 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2016-10-13 15:44:22,686 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(668): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2016-10-13 15:44:22,686 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2016-10-13 15:44:22,686 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(668): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2016-10-13 15:44:22,686 DEBUG [main-EventThread] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs 2016-10-13 15:44:22,686 DEBUG [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs 2016-10-13 15:44:22,692 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1090010022_1 at /127.0.0.1:59268 [Receiving block BP-1435415588-10.22.16.34-1476398450572:blk_1073741829_1005]] datanode.DataXceiver(252): 127.0.0.1:59228:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:59268 dst: /127.0.0.1:59228 java.io.InterruptedIOException: Interruped while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 42487 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:352) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:453) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234) at java.lang.Thread.run(Thread.java:745) 2016-10-13 15:44:22,692 ERROR [DataXceiver for client DFSClient_NONMAPREDUCE_1090010022_1 at /127.0.0.1:59271 [Receiving block BP-1435415588-10.22.16.34-1476398450572:blk_1073741831_1007]] datanode.DataXceiver(252): 127.0.0.1:59228:DataXceiver error processing WRITE_BLOCK operation src: /127.0.0.1:59271 dst: /127.0.0.1:59228 java.io.InterruptedIOException: Interruped while waiting for IO on channel java.nio.channels.SocketChannel[closed]. 41449 millis timeout left. at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:352) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:453) at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234) at java.lang.Thread.run(Thread.java:745) 2016-10-13 15:44:22,701 WARN [DataNode: [[[DISK]file:/Users/tyu/1.3-h/hbase-server/target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4/dfscluster_64f81ea3-ebf2-43ae-9724-b2d9ced5cbba/dfs/data/data1/, [DISK]file:/Users/tyu/1.3-h/hbase-server/target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4/dfscluster_64f81ea3-ebf2-43ae-9724-b2d9ced5cbba/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:59227] datanode.BPServiceActor(722): BPOfferService for Block pool BP-1435415588-10.22.16.34-1476398450572 (Datanode Uuid 08e7c6d5-66f5-4a51-9f40-6a810af4a1d6) service to localhost/127.0.0.1:59227 interrupted 2016-10-13 15:44:22,702 WARN [DataNode: [[[DISK]file:/Users/tyu/1.3-h/hbase-server/target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4/dfscluster_64f81ea3-ebf2-43ae-9724-b2d9ced5cbba/dfs/data/data1/, [DISK]file:/Users/tyu/1.3-h/hbase-server/target/test-data/cadd8616-6bf2-4a48-96e7-46f14a0665f4/dfscluster_64f81ea3-ebf2-43ae-9724-b2d9ced5cbba/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:59227] datanode.BPServiceActor(845): Ending block pool service for: Block pool BP-1435415588-10.22.16.34-1476398450572 (Datanode Uuid 08e7c6d5-66f5-4a51-9f40-6a810af4a1d6) service to localhost/127.0.0.1:59227 2016-10-13 15:44:22,725 WARN [org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor@3fae596] blockmanagement.DecommissionManager$Monitor(78): Monitor interrupted: java.lang.InterruptedException: sleep interrupted 2016-10-13 15:44:22,733 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2016-10-13 15:44:22,820 DEBUG [RS:0;10.22.16.34:59233] regionserver.HRegionServer(982): Waiting on 1588230740 2016-10-13 15:44:22,849 INFO [main] hbase.HBaseTestingUtility(1143): Minicluster is down 2016-10-13 15:44:22,849 INFO [main] regionserver.HRegionServer(1908): STOPPED: end of test 2016-10-13 15:44:22,849 INFO [RS:0;10.22.16.34:59233] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2016-10-13 15:44:22,849 INFO [RS:0;10.22.16.34:59233] regionserver.HeapMemoryManager(202): Stoping HeapMemoryTuner chore. 2016-10-13 15:44:22,849 INFO [SplitLogWorker-10.22.16.34:59233] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2016-10-13 15:44:22,849 INFO [SplitLogWorker-10.22.16.34:59233] regionserver.SplitLogWorker(155): SplitLogWorker 10.22.16.34,59233,1476398451084 exiting 2016-10-13 15:44:22,849 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2016-10-13 15:44:22,849 INFO [RS:0;10.22.16.34:59233] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2016-10-13 15:44:22,849 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2016-10-13 15:44:22,850 INFO [RS:0;10.22.16.34:59233] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2016-10-13 15:44:22,850 INFO [RS:0;10.22.16.34:59233] regionserver.HRegionServer(1056): stopping server 10.22.16.34,59233,1476398451084 2016-10-13 15:44:22,850 DEBUG [RS:0;10.22.16.34:59233] zookeeper.MetaTableLocator(622): Stopping MetaTableLocator 2016-10-13 15:44:22,850 INFO [RS:0;10.22.16.34:59233] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x157c034877e0001 2016-10-13 15:44:23,076 INFO [10.22.16.34,59233,1476398451084_ChoreService_1] hbase.ScheduledChore(183): Chore: 10.22.16.34,59233,1476398451084-MemstoreFlusherChore was stopped 2016-10-13 15:44:23,435 INFO [RS_OPEN_META-10.22.16.34:59233-0-MetaLogRoller] regionserver.LogRoller(176): LogRoller exiting. 2016-10-13 15:44:23,573 INFO [regionserver//10.22.16.34:0.logRoller] regionserver.LogRoller(176): LogRoller exiting. 2016-10-13 15:44:23,573 INFO [regionserver//10.22.16.34:0.leaseChecker] regionserver.Leases(146): regionserver//10.22.16.34:0.leaseChecker closing leases 2016-10-13 15:44:23,573 INFO [regionserver//10.22.16.34:0.leaseChecker] regionserver.Leases(149): regionserver//10.22.16.34:0.leaseChecker closed leases 2016-10-13 15:44:23,655 DEBUG [RS:0;10.22.16.34:59233] ipc.RpcClientImpl(1160): Stopping rpc client 2016-10-13 15:44:23,655 INFO [RS:0;10.22.16.34:59233] regionserver.CompactSplitThread(403): Waiting for Split Thread to finish... 2016-10-13 15:44:23,655 INFO [RS:0;10.22.16.34:59233] regionserver.CompactSplitThread(403): Waiting for Merge Thread to finish... 2016-10-13 15:44:23,655 INFO [RS:0;10.22.16.34:59233] regionserver.CompactSplitThread(403): Waiting for Large Compaction Thread to finish... 2016-10-13 15:44:23,655 INFO [RS:0;10.22.16.34:59233] regionserver.CompactSplitThread(403): Waiting for Small Compaction Thread to finish... 2016-10-13 15:44:23,655 INFO [RS:0;10.22.16.34:59233] regionserver.HRegionServer(1285): Waiting on 1 regions to close 2016-10-13 15:44:23,655 DEBUG [RS:0;10.22.16.34:59233] regionserver.HRegionServer(1289): {1588230740=hbase:meta,,1.1588230740} 2016-10-13 15:44:23,655 DEBUG [RS_CLOSE_META-10.22.16.34:59233-0] handler.CloseRegionHandler(119): Processing close of hbase:meta,,1.1588230740 2016-10-13 15:44:23,656 DEBUG [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegion(1417): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2016-10-13 15:44:23,656 DEBUG [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegion(1444): Updates disabled for region hbase:meta,,1.1588230740 2016-10-13 15:44:23,656 INFO [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegion(2252): Flushing 1/1 column families, memstore=272 B 2016-10-13 15:44:23,656 WARN [RS_OPEN_META-10.22.16.34:59233-0.append-pool15-t1] wal.FSHLog$RingBufferEventHandler(1923): Append sequenceId=8, requesting roll of WAL java.io.IOException: All datanodes 127.0.0.1:59228 are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2016-10-13 15:44:23,657 FATAL [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegionServer(2083): ABORTING region server 10.22.16.34,59233,1476398451084: Unrecoverable exception while closing region hbase:meta,,1.1588230740, still finishing close org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=8, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1925) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1777) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1699) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: All datanodes 127.0.0.1:59228 are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2016-10-13 15:44:23,657 FATAL [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegionServer(2091): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2016-10-13 15:44:23,677 INFO [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegionServer(2095): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "HeapMemoryUsage" : { "committed" : 625999872, "init" : 268435456, "max" : 2609905664, "used" : 259035984 }, "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "NonHeapMemoryUsage" : { "committed" : 83714048, "init" : 2555904, "max" : -1, "used" : 82553576 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ ], "beans" : [ ], "beans" : [ ] } 2016-10-13 15:44:23,688 ERROR [RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59249] master.MasterRpcServices(385): Region server 10.22.16.34,59233,1476398451084 reported a fatal error: ABORTING region server 10.22.16.34,59233,1476398451084: Unrecoverable exception while closing region hbase:meta,,1.1588230740, still finishing close Cause: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=8, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1925) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1777) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1699) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: All datanodes 127.0.0.1:59228 are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2016-10-13 15:44:23,695 ERROR [RS_CLOSE_META-10.22.16.34:59233-0] executor.EventHandler(228): Caught throwable while processing event M_RS_CLOSE_META java.lang.RuntimeException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=8, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:152) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=8, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1925) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1777) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1699) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) ... 3 more Caused by: java.io.IOException: All datanodes 127.0.0.1:59228 are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2016-10-13 15:44:23,695 FATAL [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegionServer(2083): ABORTING region server 10.22.16.34,59233,1476398451084: Caught throwable while processing event M_RS_CLOSE_META java.lang.RuntimeException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=8, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:152) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=8, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1925) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1777) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1699) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) ... 3 more Caused by: java.io.IOException: All datanodes 127.0.0.1:59228 are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2016-10-13 15:44:23,696 FATAL [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegionServer(2091): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2016-10-13 15:44:23,696 INFO [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegionServer(2095): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "HeapMemoryUsage" : { "committed" : 625999872, "init" : 268435456, "max" : 2609905664, "used" : 259701336 }, "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "NonHeapMemoryUsage" : { "committed" : 83714048, "init" : 2555904, "max" : -1, "used" : 82604960 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ ], "beans" : [ ], "beans" : [ ] } 2016-10-13 15:44:23,696 ERROR [RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59249] master.MasterRpcServices(385): Region server 10.22.16.34,59233,1476398451084 reported a fatal error: ABORTING region server 10.22.16.34,59233,1476398451084: Caught throwable while processing event M_RS_CLOSE_META Cause: java.lang.RuntimeException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=8, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:152) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=8, requesting roll of WAL at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1925) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1777) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1699) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) ... 3 more Caused by: java.io.IOException: All datanodes 127.0.0.1:59228 are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487) 2016-10-13 15:44:23,860 DEBUG [RS_CLOSE_META-10.22.16.34:59233-0] handler.CloseRegionHandler(119): Processing close of hbase:meta,,1.1588230740 2016-10-13 15:44:23,860 DEBUG [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegion(1417): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2016-10-13 15:44:23,860 DEBUG [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegion(1444): Updates disabled for region hbase:meta,,1.1588230740 2016-10-13 15:44:23,860 FATAL [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegionServer(2083): ABORTING region server 10.22.16.34,59233,1476398451084: Unrecoverable exception while closing region hbase:meta,,1.1588230740, still finishing close java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2195) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2180) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2144) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2134) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1462) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1381) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:138) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-10-13 15:44:23,861 FATAL [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegionServer(2091): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2016-10-13 15:44:23,861 INFO [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegionServer(2095): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "HeapMemoryUsage" : { "committed" : 625999872, "init" : 268435456, "max" : 2609905664, "used" : 260344936 }, "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "NonHeapMemoryUsage" : { "committed" : 83714048, "init" : 2555904, "max" : -1, "used" : 82616160 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ ], "beans" : [ ], "beans" : [ ] } 2016-10-13 15:44:23,862 ERROR [RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59249] master.MasterRpcServices(385): Region server 10.22.16.34,59233,1476398451084 reported a fatal error: ABORTING region server 10.22.16.34,59233,1476398451084: Unrecoverable exception while closing region hbase:meta,,1.1588230740, still finishing close Cause: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2195) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2180) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2144) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2134) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1462) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1381) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:138) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-10-13 15:44:23,862 ERROR [RS_CLOSE_META-10.22.16.34:59233-0] executor.EventHandler(228): Caught throwable while processing event M_RS_CLOSE_META java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:152) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2195) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2180) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2144) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2134) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1462) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1381) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:138) ... 4 more 2016-10-13 15:44:23,862 FATAL [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegionServer(2083): ABORTING region server 10.22.16.34,59233,1476398451084: Caught throwable while processing event M_RS_CLOSE_META java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:152) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2195) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2180) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2144) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2134) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1462) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1381) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:138) ... 4 more 2016-10-13 15:44:23,862 FATAL [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegionServer(2091): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2016-10-13 15:44:23,863 INFO [RS_CLOSE_META-10.22.16.34:59233-0] regionserver.HRegionServer(2095): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "HeapMemoryUsage" : { "committed" : 625999872, "init" : 268435456, "max" : 2609905664, "used" : 260904080 }, "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "NonHeapMemoryUsage" : { "committed" : 83714048, "init" : 2555904, "max" : -1, "used" : 82616832 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ ], "beans" : [ ], "beans" : [ ] } 2016-10-13 15:44:23,863 ERROR [RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59249] master.MasterRpcServices(385): Region server 10.22.16.34,59233,1476398451084 reported a fatal error: ABORTING region server 10.22.16.34,59233,1476398451084: Caught throwable while processing event M_RS_CLOSE_META Cause: java.lang.RuntimeException: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:152) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Aborting flush because server is aborted... at org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2195) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2180) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2144) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2134) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1462) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1381) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:138) ... 4 more 2016-10-13 15:44:24,316 DEBUG [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs 2016-10-13 15:44:24,316 DEBUG [main-EventThread] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs 2016-10-13 15:44:27,244 DEBUG [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs 2016-10-13 15:44:27,511 DEBUG [main-EventThread] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs 2016-10-13 15:44:31,736 DEBUG [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs 2016-10-13 15:44:32,147 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 30 seconds. Will retry shortly ... java.io.IOException: Failed on local exception: java.io.EOFException; Host Details : local host is: "tyus-macbook-pro.local/10.22.9.171"; destination host is: "localhost":59227; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1055) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:950) 2016-10-13 15:44:32,612 DEBUG [main-EventThread] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs 2016-10-13 15:44:33,151 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 31 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 27 more 2016-10-13 15:44:34,155 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 32 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 27 more 2016-10-13 15:44:35,162 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 33 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 27 more 2016-10-13 15:44:36,166 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 34 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:37,171 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 35 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:38,174 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 36 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:39,181 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 37 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:40,188 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 38 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:40,698 DEBUG [main-EventThread] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs 2016-10-13 15:44:40,699 ERROR [main-EventThread] zookeeper.RecoverableZooKeeper(274): ZooKeeper getChildren failed after 4 attempts 2016-10-13 15:44:40,699 WARN [main-EventThread] zookeeper.ZKUtil(464): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Unable to list children of znode /hbase/rs org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:292) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:457) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:485) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl.getRegisteredRegionServers(ReplicationTrackerZKImpl.java:245) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl.refreshOtherRegionServersList(ReplicationTrackerZKImpl.java:226) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl.access$400(ReplicationTrackerZKImpl.java:42) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher.refreshListIfRightPath(ReplicationTrackerZKImpl.java:142) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher.nodeChildrenChanged(ReplicationTrackerZKImpl.java:135) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:638) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2016-10-13 15:44:40,699 ERROR [main-EventThread] zookeeper.ZooKeeperWatcher(719): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:292) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:457) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:485) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl.getRegisteredRegionServers(ReplicationTrackerZKImpl.java:245) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl.refreshOtherRegionServersList(ReplicationTrackerZKImpl.java:226) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl.access$400(ReplicationTrackerZKImpl.java:42) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher.refreshListIfRightPath(ReplicationTrackerZKImpl.java:142) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher.nodeChildrenChanged(ReplicationTrackerZKImpl.java:135) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:638) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2016-10-13 15:44:40,699 FATAL [main-EventThread] regionserver.HRegionServer(2083): ABORTING region server 10.22.16.34,59233,1476398451084: Get list of registered region servers org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:292) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:457) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:485) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl.getRegisteredRegionServers(ReplicationTrackerZKImpl.java:245) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl.refreshOtherRegionServersList(ReplicationTrackerZKImpl.java:226) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl.access$400(ReplicationTrackerZKImpl.java:42) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher.refreshListIfRightPath(ReplicationTrackerZKImpl.java:142) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher.nodeChildrenChanged(ReplicationTrackerZKImpl.java:135) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:638) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2016-10-13 15:44:40,699 FATAL [main-EventThread] regionserver.HRegionServer(2091): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2016-10-13 15:44:40,700 INFO [main-EventThread] regionserver.HRegionServer(2095): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "HeapMemoryUsage" : { "committed" : 625999872, "init" : 268435456, "max" : 2609905664, "used" : 264413664 }, "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "NonHeapMemoryUsage" : { "committed" : 83845120, "init" : 2555904, "max" : -1, "used" : 82776512 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ ], "beans" : [ ], "beans" : [ ] } 2016-10-13 15:44:40,700 ERROR [RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59249] master.MasterRpcServices(385): Region server 10.22.16.34,59233,1476398451084 reported a fatal error: ABORTING region server 10.22.16.34,59233,1476398451084: Get list of registered region servers Cause: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:292) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:457) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:485) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl.getRegisteredRegionServers(ReplicationTrackerZKImpl.java:245) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl.refreshOtherRegionServersList(ReplicationTrackerZKImpl.java:226) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl.access$400(ReplicationTrackerZKImpl.java:42) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher.refreshListIfRightPath(ReplicationTrackerZKImpl.java:142) at org.apache.hadoop.hbase.replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher.nodeChildrenChanged(ReplicationTrackerZKImpl.java:135) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:638) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) 2016-10-13 15:44:40,700 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2016-10-13 15:44:40,701 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(668): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2016-10-13 15:44:40,701 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(600): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2016-10-13 15:44:40,701 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(668): regionserver:59233-0x157c034877e0000, quorum=localhost:55969, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2016-10-13 15:44:41,192 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 39 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:41,379 DEBUG [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs 2016-10-13 15:44:41,380 ERROR [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(274): ZooKeeper getChildren failed after 4 attempts 2016-10-13 15:44:41,380 WARN [M:0;10.22.16.34:59249] master.ServerManager(579): Failed to list regionservers org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:292) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenNoWatch(ZKUtil.java:514) at org.apache.hadoop.hbase.master.ServerManager.letRegionServersShutdown(ServerManager.java:571) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1178) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1108) at java.lang.Thread.run(Thread.java:745) 2016-10-13 15:44:41,380 DEBUG [M:0;10.22.16.34:59249] master.HMaster(1181): Stopping service threads 2016-10-13 15:44:42,197 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 40 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:42,698 DEBUG [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master 2016-10-13 15:44:42,902 INFO [main] hbase.ResourceChecker(171): after: regionserver.TestClusterId#testClusterId Thread=90 (was 19) Potentially hanging thread: JvmPauseMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: region-location-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.FifoWFPBQ.priority.handler=0,queue=0,port=59233 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.reader=1,bindAddress=10.22.16.34,port=59233 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:685) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:672) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.FifoWFPBQ.priority.handler=1,queue=1,port=59233 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.FifoWFPBQ.replication.handler=0,queue=0,port=59249 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: 10.22.16.34,59233,1476398451084_ChoreService_1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS:0;10.22.16.34:59233.append-pool18-t1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45) com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55) com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=0,bindAddress=10.22.16.34,port=59249 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:685) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:672) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.FifoWFPBQ.priority.handler=2,queue=0,port=59233 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: M:0;10.22.16.34:59249 java.lang.Thread.sleep(Native Method) java.lang.Thread.sleep(Thread.java:340) java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386) org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:158) org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:367) org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:624) org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148) org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267) org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1188) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1108) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS_OPEN_META-10.22.16.34:59233-0.append-pool15-t1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:45) com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55) com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS_OPEN_META-10.22.16.34:59233-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: sync.2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: LeaseRenewer:tyu.hfs.1@localhost:59227 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=59249 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RS:0;10.22.16.34:59233 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1327) org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1313) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1083) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59249 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: region-location-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=2,bindAddress=10.22.16.34,port=59249 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:685) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:672) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: main-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: sync.1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.FifoWFPBQ.priority.handler=1,queue=1,port=59249 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.FifoWFPBQ.default.handler=1,queue=0,port=59233 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.FifoWFPBQ.default.handler=1,queue=0,port=59249 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=59233 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: 10.22.16.34:59249.activeMasterManager-SendThread(localhost:55969) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) Potentially hanging thread: 10.22.16.34,59249,1476398461537_splitLogManager__ChoreService_1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.FifoWFPBQ.replication.handler=0,queue=0,port=59233 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.FifoWFPBQ.replication.handler=2,queue=0,port=59249 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.FifoWFPBQ.priority.handler=4,queue=0,port=59249 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.FifoWFPBQ.priority.handler=3,queue=1,port=59233 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=59233 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: sync.1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: sync.3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: sync.2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.FifoWFPBQ.priority.handler=3,queue=1,port=59249 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS_CLOSE_META-10.22.16.34:59233-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.22.16.34:59249.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: Timer for 'DataNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RpcServer.FifoWFPBQ.replication.handler=1,queue=0,port=59233 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.FifoWFPBQ.replication.handler=2,queue=0,port=59233 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=59249 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.listener,port=59233 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:791) Potentially hanging thread: IPC Client (1711693104) connection to /10.22.16.34:59249 from tyu java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.waitForWork(RpcClientImpl.java:550) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:565) Potentially hanging thread: Thread-240 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.HMaster$InitializationMonitor.run(HMaster.java:231) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.FifoWFPBQ.priority.handler=2,queue=0,port=59249 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: sync.3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: main-SendThread(localhost:55969) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: 10.22.16.34:59233Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: LeaseRenewer:tyu@localhost:59227 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59249 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.FifoWFPBQ.replication.handler=1,queue=0,port=59249 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: sync.4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.FifoWFPBQ.default.handler=2,queue=0,port=59249 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: region-location-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: sync.4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.22.16.34:59249.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: RpcServer.FifoWFPBQ.priority.handler=0,queue=0,port=59249 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: sync.0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.FifoWFPBQ.default.handler=2,queue=0,port=59233 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.responder sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRunLoop(RpcServer.java:1031) org.apache.hadoop.hbase.ipc.RpcServer$Responder.run(RpcServer.java:978) Potentially hanging thread: RpcServer.reader=0,bindAddress=10.22.16.34,port=59233 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:685) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:672) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.22.16.34:59249.activeMasterManager java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:101) org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1014) org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:806) org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:195) org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1827) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.FifoWFPBQ.priority.handler=4,queue=0,port=59233 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Potentially hanging thread: RpcServer.responder sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRunLoop(RpcServer.java:1031) org.apache.hadoop.hbase.ipc.RpcServer$Responder.run(RpcServer.java:978) Potentially hanging thread: RS_COMPACTED_FILES_DISCHARGER-10.22.16.34:59233-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:651) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.reader=1,bindAddress=10.22.16.34,port=59249 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:685) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:672) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: main-SendThread(localhost:55969) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: main-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: RpcServer.reader=2,bindAddress=10.22.16.34,port=59233 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:685) org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:672) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RpcServer.listener,port=59249 sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:791) Potentially hanging thread: sync.0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.run(FSHLog.java:1231) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: 10.22.16.34:59249.activeMasterManager-SendThread(localhost:55969) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) Potentially hanging thread: RpcServer.FifoWFPBQ.default.handler=4,queue=0,port=59233 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) - Thread LEAK? -, OpenFileDescriptor=234 (was 214) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=163 (was 309), ProcessCount=283 (was 284), AvailableMemoryMB=274 (was 107) - AvailableMemoryMB LEAK? -, ConnectionCount=0 (was 0) 2016-10-13 15:44:42,906 INFO [Thread-2] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@3427b02d 2016-10-13 15:44:43,203 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 41 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:44,000 DEBUG [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master 2016-10-13 15:44:44,210 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 42 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:45,214 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 43 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:46,221 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 44 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:47,122 DEBUG [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master 2016-10-13 15:44:47,228 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 45 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:48,233 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 46 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:49,239 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 47 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:50,246 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 48 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:51,253 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 49 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:52,100 DEBUG [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master 2016-10-13 15:44:52,257 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 50 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:52,638 INFO [10.22.16.34,59233,1476398451084_ChoreService_1] hbase.ScheduledChore(183): Chore: CompactedHFilesCleaner was stopped 2016-10-13 15:44:53,263 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 51 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:54,268 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 52 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:55,203 INFO [ReplicationExecutor-0] regionserver.ReplicationSourceManager$NodeFailoverWorker(658): Not transferring queue since we are shutting down 2016-10-13 15:44:55,271 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 53 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:56,277 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 54 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:57,283 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 55 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:58,288 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 56 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:44:59,292 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 57 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:45:00,299 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 58 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:45:01,305 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 59 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:45:01,565 DEBUG [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master 2016-10-13 15:45:01,565 ERROR [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(274): ZooKeeper getData failed after 4 attempts 2016-10-13 15:45:01,565 WARN [M:0;10.22.16.34:59249] zookeeper.ZKUtil(632): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Unable to get data of znode /hbase/master org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:624) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1188) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1108) at java.lang.Thread.run(Thread.java:745) 2016-10-13 15:45:01,565 ERROR [M:0;10.22.16.34:59249] zookeeper.ZooKeeperWatcher(719): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:624) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1188) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1108) at java.lang.Thread.run(Thread.java:745) 2016-10-13 15:45:01,565 ERROR [M:0;10.22.16.34:59249] master.ActiveMasterManager(278): master:59249-0x157c034877e0002, quorum=localhost:55969, baseZNode=/hbase Error deleting our own master address node org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:624) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1188) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1108) at java.lang.Thread.run(Thread.java:745) 2016-10-13 15:45:01,565 INFO [M:0;10.22.16.34:59249] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x157c034877e0006 2016-10-13 15:45:01,987 DEBUG [M:0;10.22.16.34:59249] ipc.RpcClientImpl(1160): Stopping rpc client 2016-10-13 15:45:01,987 INFO [M:0;10.22.16.34:59249] hbase.ChoreService(328): Chore service for: 10.22.16.34,59249,1476398461537_splitLogManager_ had [] on shutdown 2016-10-13 15:45:01,987 INFO [M:0;10.22.16.34:59249] flush.MasterFlushTableProcedureManager(79): stop: server shutting down. 2016-10-13 15:45:01,987 INFO [M:0;10.22.16.34:59249] ipc.RpcServer(2360): Stopping server on 59249 2016-10-13 15:45:01,987 INFO [RpcServer.listener,port=59249] ipc.RpcServer$Listener(834): RpcServer.listener,port=59249: stopping 2016-10-13 15:45:01,987 INFO [RpcServer.responder] ipc.RpcServer$Responder(1077): RpcServer.responder: stopped 2016-10-13 15:45:01,988 INFO [RpcServer.responder] ipc.RpcServer$Responder(980): RpcServer.responder: stopping 2016-10-13 15:45:02,310 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 60 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:45:03,072 INFO [10.22.16.34,59233,1476398451084_ChoreService_1] hbase.ScheduledChore(183): Chore: MovedRegionsCleaner for region 10.22.16.34,59233,1476398451084 was stopped 2016-10-13 15:45:03,317 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 61 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:45:03,337 DEBUG [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs/10.22.16.34,59249,1476398461537 2016-10-13 15:45:04,321 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 62 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:45:05,287 DEBUG [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs/10.22.16.34,59249,1476398461537 2016-10-13 15:45:05,326 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 63 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:45:06,332 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 64 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:45:07,340 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 65 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:45:08,345 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 66 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:45:08,700 DEBUG [M:0;10.22.16.34:59249] zookeeper.RecoverableZooKeeper(272): Possibly transient ZooKeeper, quorum=localhost:55969, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs/10.22.16.34,59249,1476398461537 2016-10-13 15:45:09,351 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 67 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:45:10,358 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 68 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:45:11,362 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 69 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more 2016-10-13 15:45:12,368 WARN [LeaseRenewer:tyu@localhost:59227] hdfs.LeaseRenewer(458): Failed to renew lease for [DFSClient_NONMAPREDUCE_1090010022_1] for 70 seconds. Will retry shortly ... java.net.ConnectException: Call From tyus-macbook-pro.local/10.22.9.171 to localhost:59227 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor22.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1415) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy23.renewLease(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:302) at com.sun.proxy.$Proxy24.renewLease(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814) at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442) at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) at org.apache.hadoop.ipc.Client.call(Client.java:1382) ... 26 more