Skip to content  log in   • ENABLE AUTO REFRESH • Jenkins • hbase2-release • #80 • Test Results • Test Results • org.apache.hadoop.hbase.regionserver • TestSplitTransactionOnCluster • testMasterRestartAtRegionSplitPendingCatalogJanitor •  Back to Project  Status  Changes  Console Output  View as plain text  View Build Information  ????  ????  Timings  Environment Variables  Git Build Data  FindBugs Warnings  Test Result  ?????  ????? Failed org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testMasterRestartAtRegionSplitPendingCatalogJanitor Failing for the past 1 build (Since #80 ) ????:13 ? Error Message expected: but was: Stacktrace java.lang.AssertionError: expected: but was: at org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testMasterRestartAtRegionSplitPendingCatalogJanitor(TestSplitTransactionOnCluster.java:577) Standard Output java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Standard Error 2019-07-23 18:33:51,750 INFO [Time-limited test] hbase.ResourceChecker(172): after: regionserver.TestSplitTransactionOnCluster#testShutdownFixupWhenDaughterHasSplit Thread=392 (was 403), OpenFileDescriptor=641 (was 629) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=1048576 (was 1048576), SystemLoadAverage=990 (was 1113), ProcessCount=19 (was 19), AvailableMemoryMB=18733 (was 18229) - AvailableMemoryMB LEAK? - 2019-07-23 18:33:51,847 INFO [Time-limited test] hbase.ResourceChecker(148): before: regionserver.TestSplitTransactionOnCluster#testMasterRestartAtRegionSplitPendingCatalogJanitor Thread=392, OpenFileDescriptor=641, MaxFileDescriptor=1048576, SystemLoadAverage=990, ProcessCount=18, AvailableMemoryMB=18730 2019-07-23 18:33:51,848 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:0;maven-jdk8-551kz:43427 2019-07-23 18:33:52,037 INFO [Time-limited test] client.ConnectionUtils(129): regionserver/maven-jdk8-551kz:0 server-side Connection retries=45 2019-07-23 18:33:52,038 INFO [Time-limited test] ipc.RpcExecutor(150): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2019-07-23 18:33:52,038 INFO [Time-limited test] ipc.RpcExecutor(150): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=6 2019-07-23 18:33:52,038 INFO [Time-limited test] ipc.RpcExecutor(150): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=3 2019-07-23 18:33:52,038 INFO [Time-limited test] ipc.RpcExecutor(150): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=1 2019-07-23 18:33:52,039 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2019-07-23 18:33:52,039 INFO [Time-limited test] io.ByteBufferPool(83): Created with bufferSize=64 KB and maxPoolSize=320 B 2019-07-23 18:33:52,041 INFO [Time-limited test] ipc.NettyRpcServer(115): Bind to /10.119.14.43:40727 2019-07-23 18:33:52,042 INFO [Time-limited test] hfile.BlockCacheFactory(116): Allocating onheap LruBlockCache size=1.07 GB, blockSize=64 KB 2019-07-23 18:33:52,043 DEBUG [Time-limited test] mob.MobFileCache(124): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2019-07-23 18:33:52,045 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2019-07-23 18:33:52,047 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2019-07-23 18:33:52,048 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(101): Process identifier=regionserver:40727 connecting to ZooKeeper ensemble=localhost:62549 2019-07-23 18:33:52,052 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(484): regionserver:407270x0, quorum=localhost:62549, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2019-07-23 18:33:52,052 DEBUG [Time-limited test] zookeeper.ZKUtil(359): regionserver:407270x0, quorum=localhost:62549, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2019-07-23 18:33:52,053 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(549): regionserver:40727-0x16c1e642279000a connected 2019-07-23 18:33:52,135 DEBUG [Time-limited test] zookeeper.ZKUtil(359): regionserver:40727-0x16c1e642279000a, quorum=localhost:62549, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2019-07-23 18:33:52,135 DEBUG [Time-limited test] zookeeper.ZKUtil(361): regionserver:40727-0x16c1e642279000a, quorum=localhost:62549, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2019-07-23 18:33:52,136 DEBUG [Time-limited test] ipc.RpcExecutor(277): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40727 2019-07-23 18:33:52,138 DEBUG [Time-limited test] ipc.RpcExecutor(277): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=40727 2019-07-23 18:33:52,138 DEBUG [Time-limited test] ipc.RpcExecutor(277): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40727 2019-07-23 18:33:52,139 DEBUG [Time-limited test] ipc.RpcExecutor(277): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40727 2019-07-23 18:33:52,143 INFO [RS:3;maven-jdk8-551kz:40727] regionserver.HRegionServer(914): ClusterId : 061e4cab-8f8e-41ff-9820-cd30061d5282 2019-07-23 18:33:52,144 DEBUG [RS:3;maven-jdk8-551kz:40727] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initializing 2019-07-23 18:33:52,146 DEBUG [RS:3;maven-jdk8-551kz:40727] procedure.RegionServerProcedureManagerHost(47): Procedure flush-table-proc initialized 2019-07-23 18:33:52,146 DEBUG [RS:3;maven-jdk8-551kz:40727] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initializing 2019-07-23 18:33:52,147 DEBUG [RS:3;maven-jdk8-551kz:40727] procedure.RegionServerProcedureManagerHost(47): Procedure online-snapshot initialized 2019-07-23 18:33:52,149 INFO [RS:3;maven-jdk8-551kz:40727] zookeeper.ReadOnlyZKClient(142): Connect 0x2cd60f64 to localhost:62549 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2019-07-23 18:33:52,240 DEBUG [RS:3;maven-jdk8-551kz:40727] ipc.AbstractRpcClient(210): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2211f48e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2019-07-23 18:33:52,241 DEBUG [RS:3;maven-jdk8-551kz:40727] ipc.AbstractRpcClient(210): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@632cfeb5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=maven-jdk8-551kz/10.119.14.43:0 2019-07-23 18:33:52,241 DEBUG [RS:3;maven-jdk8-551kz:40727] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:3;maven-jdk8-551kz:40727 2019-07-23 18:33:52,241 INFO [RS:3;maven-jdk8-551kz:40727] regionserver.RegionServerCoprocessorHost(67): System coprocessor loading is enabled 2019-07-23 18:33:52,241 INFO [RS:3;maven-jdk8-551kz:40727] regionserver.RegionServerCoprocessorHost(68): Table coprocessor loading is enabled 2019-07-23 18:33:52,241 INFO [RS:3;maven-jdk8-551kz:40727] regionserver.HRegionServer(2669): reportForDuty to master=maven-jdk8-551kz,46015,1563877977939 with port=40727, startcode=1563878032037 2019-07-23 18:33:52,248 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(556): Connection from 10.119.14.43:46591, version=2.2.0-mdh3.1.1, sasl=false, ugi=default.hfs.3 (auth:SIMPLE), service=RegionServerStatusService 2019-07-23 18:33:52,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] master.ServerManager(402): Registering regionserver=maven-jdk8-551kz,40727,1563878032037 2019-07-23 18:33:52,335 DEBUG [RS:3;maven-jdk8-551kz:40727] regionserver.HRegionServer(1543): Config from master: hbase.rootdir=hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f 2019-07-23 18:33:52,336 DEBUG [RS:3;maven-jdk8-551kz:40727] regionserver.HRegionServer(1543): Config from master: fs.defaultFS=hdfs://localhost:46863 2019-07-23 18:33:52,336 DEBUG [RS:3;maven-jdk8-551kz:40727] regionserver.HRegionServer(1543): Config from master: hbase.master.info.port=-1 2019-07-23 18:33:52,337 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(484): regionserver:45127-0x16c1e6422790003, quorum=localhost:62549, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2019-07-23 18:33:52,337 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(484): regionserver:39239-0x16c1e6422790002, quorum=localhost:62549, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2019-07-23 18:33:52,337 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(484): master:46015-0x16c1e6422790000, quorum=localhost:62549, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2019-07-23 18:33:52,337 DEBUG [RS:3;maven-jdk8-551kz:40727] zookeeper.ZKUtil(359): regionserver:40727-0x16c1e642279000a, quorum=localhost:62549, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/maven-jdk8-551kz,40727,1563878032037 2019-07-23 18:33:52,338 WARN [RS:3;maven-jdk8-551kz:40727] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2019-07-23 18:33:52,338 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(359): regionserver:39239-0x16c1e6422790002, quorum=localhost:62549, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/maven-jdk8-551kz,45127,1563877982046 2019-07-23 18:33:52,338 INFO [RS:3;maven-jdk8-551kz:40727] wal.WALFactory(157): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2019-07-23 18:33:52,338 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(359): regionserver:45127-0x16c1e6422790003, quorum=localhost:62549, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/maven-jdk8-551kz,45127,1563877982046 2019-07-23 18:33:52,338 INFO [RegionServerTracker-0] master.RegionServerTracker(182): RegionServer ephemeral node created, adding [maven-jdk8-551kz,40727,1563878032037] 2019-07-23 18:33:52,338 DEBUG [RS:3;maven-jdk8-551kz:40727] regionserver.HRegionServer(1865): logDir=hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/WALs/maven-jdk8-551kz,40727,1563878032037 2019-07-23 18:33:52,338 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(359): regionserver:39239-0x16c1e6422790002, quorum=localhost:62549, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/maven-jdk8-551kz,39239,1563877981752 2019-07-23 18:33:52,338 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(359): regionserver:45127-0x16c1e6422790003, quorum=localhost:62549, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/maven-jdk8-551kz,39239,1563877981752 2019-07-23 18:33:52,339 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(359): regionserver:45127-0x16c1e6422790003, quorum=localhost:62549, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/maven-jdk8-551kz,40727,1563878032037 2019-07-23 18:33:52,339 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(359): regionserver:39239-0x16c1e6422790002, quorum=localhost:62549, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/maven-jdk8-551kz,40727,1563878032037 2019-07-23 18:33:52,347 DEBUG [RS:3;maven-jdk8-551kz:40727] zookeeper.ZKUtil(359): regionserver:40727-0x16c1e642279000a, quorum=localhost:62549, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/maven-jdk8-551kz,45127,1563877982046 2019-07-23 18:33:52,347 DEBUG [RS:3;maven-jdk8-551kz:40727] zookeeper.ZKUtil(359): regionserver:40727-0x16c1e642279000a, quorum=localhost:62549, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/maven-jdk8-551kz,39239,1563877981752 2019-07-23 18:33:52,348 DEBUG [RS:3;maven-jdk8-551kz:40727] zookeeper.ZKUtil(359): regionserver:40727-0x16c1e642279000a, quorum=localhost:62549, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/maven-jdk8-551kz,40727,1563878032037 2019-07-23 18:33:52,350 DEBUG [RS:3;maven-jdk8-551kz:40727] regionserver.Replication(154): Replication stats-in-log period=300 seconds 2019-07-23 18:33:52,350 INFO [RS:3;maven-jdk8-551kz:40727] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2019-07-23 18:33:52,435 INFO [RS:3;maven-jdk8-551kz:40727] regionserver.MemStoreFlusher(132): globalMemStoreLimit=1.1 G, globalMemStoreLimitLowMark=1.0 G, Offheap=false 2019-07-23 18:33:52,436 INFO [RS:3;maven-jdk8-551kz:40727] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2019-07-23 18:33:52,436 INFO [RS:3;maven-jdk8-551kz:40727] regionserver.HRegionServer$CompactionChecker(1751): CompactionChecker runs every PT1S 2019-07-23 18:33:52,436 INFO [RS:3;maven-jdk8-551kz:40727] ipc.QueueFullDetector(137): QueueFullDetector is started, denseCheckNum=100, densePeriod=1000, sampleThreshold=95, rejectThreshold=95 2019-07-23 18:33:52,440 DEBUG [RS:3;maven-jdk8-551kz:40727] executor.ExecutorService(99): Starting executor service name=RS_OPEN_REGION-regionserver/maven-jdk8-551kz:0, corePoolSize=3, maxPoolSize=3 2019-07-23 18:33:52,440 DEBUG [RS:3;maven-jdk8-551kz:40727] executor.ExecutorService(99): Starting executor service name=RS_OPEN_META-regionserver/maven-jdk8-551kz:0, corePoolSize=1, maxPoolSize=1 2019-07-23 18:33:52,440 DEBUG [RS:3;maven-jdk8-551kz:40727] executor.ExecutorService(99): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/maven-jdk8-551kz:0, corePoolSize=3, maxPoolSize=3 2019-07-23 18:33:52,440 DEBUG [RS:3;maven-jdk8-551kz:40727] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_REGION-regionserver/maven-jdk8-551kz:0, corePoolSize=3, maxPoolSize=3 2019-07-23 18:33:52,440 DEBUG [RS:3;maven-jdk8-551kz:40727] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_META-regionserver/maven-jdk8-551kz:0, corePoolSize=1, maxPoolSize=1 2019-07-23 18:33:52,441 DEBUG [RS:3;maven-jdk8-551kz:40727] executor.ExecutorService(99): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/maven-jdk8-551kz:0, corePoolSize=2, maxPoolSize=2 2019-07-23 18:33:52,441 DEBUG [RS:3;maven-jdk8-551kz:40727] executor.ExecutorService(99): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/maven-jdk8-551kz:0, corePoolSize=10, maxPoolSize=10 2019-07-23 18:33:52,441 DEBUG [RS:3;maven-jdk8-551kz:40727] executor.ExecutorService(99): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/maven-jdk8-551kz:0, corePoolSize=3, maxPoolSize=3 2019-07-23 18:33:52,441 DEBUG [RS:3;maven-jdk8-551kz:40727] executor.ExecutorService(99): Starting executor service name=RS_REFRESH_PEER-regionserver/maven-jdk8-551kz:0, corePoolSize=2, maxPoolSize=2 2019-07-23 18:33:52,441 DEBUG [RS:3;maven-jdk8-551kz:40727] executor.ExecutorService(99): Starting executor service name=RS_REPLAY_SYNC_REPLICATION_WAL-regionserver/maven-jdk8-551kz:0, corePoolSize=1, maxPoolSize=1 2019-07-23 18:33:52,441 DEBUG [RS:3;maven-jdk8-551kz:40727] executor.ExecutorService(99): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/maven-jdk8-551kz:0, corePoolSize=1, maxPoolSize=1 2019-07-23 18:33:52,639 INFO [RS:3;maven-jdk8-551kz:40727] regionserver.HeapMemoryManager(210): Starting, tuneOn=false 2019-07-23 18:33:52,640 INFO [SplitLogWorker-maven-jdk8-551kz:40727] regionserver.SplitLogWorker(213): SplitLogWorker maven-jdk8-551kz,40727,1563878032037 starting 2019-07-23 18:33:52,843 INFO [RS:3;maven-jdk8-551kz:40727] regionserver.HRegionServer(1584): Serving as maven-jdk8-551kz,40727,1563878032037, RpcServer on maven-jdk8-551kz/10.119.14.43:40727, sessionid=0x16c1e642279000a 2019-07-23 18:33:52,843 INFO [Time-limited test] hbase.HBaseTestingUtility(3364): Started new server=Thread[RS:3;maven-jdk8-551kz:40727,5,FailOnTimeoutGroup] 2019-07-23 18:33:52,844 DEBUG [RS:3;maven-jdk8-551kz:40727] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc starting 2019-07-23 18:33:52,844 DEBUG [RS:3;maven-jdk8-551kz:40727] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager maven-jdk8-551kz,40727,1563878032037 2019-07-23 18:33:52,844 INFO [Time-limited test] hbase.HBaseTestingUtility(3390): A region server is stopped or stopping:maven-jdk8-551kz,43427,1563877981440 2019-07-23 18:33:52,844 DEBUG [RS:3;maven-jdk8-551kz:40727] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'maven-jdk8-551kz,40727,1563878032037' 2019-07-23 18:33:52,844 DEBUG [RS:3;maven-jdk8-551kz:40727] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-07-23 18:33:52,845 DEBUG [RS:3;maven-jdk8-551kz:40727] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-07-23 18:33:52,845 DEBUG [RS:3;maven-jdk8-551kz:40727] procedure.RegionServerProcedureManagerHost(55): Procedure flush-table-proc started 2019-07-23 18:33:52,845 DEBUG [RS:3;maven-jdk8-551kz:40727] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot starting 2019-07-23 18:33:52,845 DEBUG [RS:3;maven-jdk8-551kz:40727] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager maven-jdk8-551kz,40727,1563878032037 2019-07-23 18:33:52,845 DEBUG [RS:3;maven-jdk8-551kz:40727] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'maven-jdk8-551kz,40727,1563878032037' 2019-07-23 18:33:52,845 DEBUG [RS:3;maven-jdk8-551kz:40727] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2019-07-23 18:33:52,845 DEBUG [RS:3;maven-jdk8-551kz:40727] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2019-07-23 18:33:52,846 DEBUG [RS:3;maven-jdk8-551kz:40727] procedure.RegionServerProcedureManagerHost(55): Procedure online-snapshot started 2019-07-23 18:33:52,846 INFO [RS:3;maven-jdk8-551kz:40727] quotas.RegionServerRpcQuotaManager(67): Quota support disabled 2019-07-23 18:33:52,846 INFO [RS:3;maven-jdk8-551kz:40727] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2019-07-23 18:33:52,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] master.HMaster$4(2048): Client=default//10.119.14.43 create 'testMasterRestartAtRegionSplitPendingCatalogJanitor', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}, {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2019-07-23 18:33:52,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] procedure2.ProcedureExecutor(1027): Stored pid=39, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:33:52,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=39, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:33:52,951 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=39, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:33:52,951 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (39) sharedLock=0 size=0) from run queue because: pid=39, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor held the exclusive lock 2019-07-23 18:33:52,954 DEBUG [PEWorker-15] procedure2.RootProcedureState(153): Add procedure pid=39, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 0th rollback step 2019-07-23 18:33:52,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] master.MasterRpcServices(683): Client=default//10.119.14.43 procedure request for creating table: namespace: "default" qualifier: "testMasterRestartAtRegionSplitPendingCatalogJanitor" procId is: 39 2019-07-23 18:33:52,955 WARN [Default-IPC-NioEventLoopGroup-13-2] ipc.AbstractRpcClient(402): Slow ipc call, MethodName=CreateTable, consume time=109, remote address:maven-jdk8-551kz/10.119.14.43:46015 2019-07-23 18:33:52,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] master.MasterRpcServices(1253): Checking to see if procedure is done pid=39 2019-07-23 18:33:52,956 DEBUG [HFileArchiver-5] backup.HFileArchiver(129): ARCHIVING hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/6c32927df5feade75feea233101e7422 2019-07-23 18:33:52,957 DEBUG [HFileArchiver-5] backup.HFileArchiver(152): Directory hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/6c32927df5feade75feea233101e7422 empty. 2019-07-23 18:33:52,958 DEBUG [HFileArchiver-5] backup.HFileArchiver(529): Failed to delete directory hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/6c32927df5feade75feea233101e7422 2019-07-23 18:33:52,958 DEBUG [PEWorker-15] procedure.DeleteTableProcedure(327): Table 'testMasterRestartAtRegionSplitPendingCatalogJanitor' archived! 2019-07-23 18:33:53,037 INFO [IPC Server handler 3 on 46863] blockmanagement.BlockManager(2599): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43977 is added to blk_1073741853_1029{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction, ReplicaUnderConstruction, ReplicaUnderConstruction]} size 0 2019-07-23 18:33:53,038 INFO [IPC Server handler 4 on 46863] blockmanagement.BlockManager(2599): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42815 is added to blk_1073741853_1029{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction, ReplicaUnderConstruction, ReplicaUnderConstruction]} size 0 2019-07-23 18:33:53,038 INFO [IPC Server handler 5 on 46863] blockmanagement.BlockManager(2599): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41521 is added to blk_1073741853_1029{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction, ReplicaUnderConstruction, ReplicaUnderConstruction]} size 0 2019-07-23 18:33:53,040 DEBUG [PEWorker-15] util.FSTableDescriptors(695): Wrote into hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/.tabledesc/.tableinfo.0000000001 2019-07-23 18:33:53,042 INFO [RegionOpenAndInitThread-testMasterRestartAtRegionSplitPendingCatalogJanitor-1] regionserver.HRegion(7237): creating HRegion testMasterRestartAtRegionSplitPendingCatalogJanitor HTD == 'testMasterRestartAtRegionSplitPendingCatalogJanitor', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}, {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp Table name == testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:33:53,051 INFO [IPC Server handler 4 on 46863] blockmanagement.BlockManager(2599): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43977 is added to blk_1073741854_1030{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction, ReplicaUnderConstruction, ReplicaUnderConstruction]} size 0 2019-07-23 18:33:53,052 INFO [IPC Server handler 5 on 46863] blockmanagement.BlockManager(2599): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42815 is added to blk_1073741854_1030{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction, ReplicaUnderConstruction, ReplicaUnderConstruction]} size 0 2019-07-23 18:33:53,052 INFO [IPC Server handler 6 on 46863] blockmanagement.BlockManager(2599): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41521 is added to blk_1073741854_1030 size 86 2019-07-23 18:33:53,052 DEBUG [RegionOpenAndInitThread-testMasterRestartAtRegionSplitPendingCatalogJanitor-1] regionserver.HRegion(868): Instantiated testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878032846.6c32927df5feade75feea233101e7422.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2019-07-23 18:33:53,053 DEBUG [RegionOpenAndInitThread-testMasterRestartAtRegionSplitPendingCatalogJanitor-1] regionserver.HRegion(1657): Closing 6c32927df5feade75feea233101e7422, disabling compactions & flushes 2019-07-23 18:33:53,053 DEBUG [RegionOpenAndInitThread-testMasterRestartAtRegionSplitPendingCatalogJanitor-1] regionserver.HRegion(1697): Updates disabled for region testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878032846.6c32927df5feade75feea233101e7422. 2019-07-23 18:33:53,053 INFO [RegionOpenAndInitThread-testMasterRestartAtRegionSplitPendingCatalogJanitor-1] regionserver.HRegion(1814): Closed testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878032846.6c32927df5feade75feea233101e7422. 2019-07-23 18:33:53,055 DEBUG [PEWorker-15] procedure2.RootProcedureState(153): Add procedure pid=39, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 1th rollback step 2019-07-23 18:33:53,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] master.MasterRpcServices(1253): Checking to see if procedure is done pid=39 2019-07-23 18:33:53,058 DEBUG [PEWorker-15] hbase.MetaTableAccessor(2138): Put {"totalColumns":2,"row":"testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878032846.6c32927df5feade75feea233101e7422.","families":{"info":[{"qualifier":"regioninfo","vlen":85,"tag":[],"timestamp":"1563878033058"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1563878033058"}]},"ts":"1563878033058"} 2019-07-23 18:33:53,061 INFO [PEWorker-15] hbase.MetaTableAccessor(1528): Added 1 regions to meta. 2019-07-23 18:33:53,061 DEBUG [PEWorker-15] procedure2.RootProcedureState(153): Add procedure pid=39, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 2th rollback step 2019-07-23 18:33:53,063 DEBUG [PEWorker-15] hbase.MetaTableAccessor(2138): Put {"totalColumns":1,"row":"testMasterRestartAtRegionSplitPendingCatalogJanitor","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1563878033062"}]},"ts":"1563878033062"} 2019-07-23 18:33:53,065 INFO [PEWorker-15] hbase.MetaTableAccessor(1673): Updated tableName=testMasterRestartAtRegionSplitPendingCatalogJanitor, state=ENABLING in hbase:meta 2019-07-23 18:33:53,141 INFO [PEWorker-15] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN}] 2019-07-23 18:33:53,141 DEBUG [PEWorker-15] procedure2.RootProcedureState(153): Add procedure pid=39, state=WAITING:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 3th rollback step 2019-07-23 18:33:53,143 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (39) sharedLock=0 size=1) to run queue because: pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN has the excusive lock access 2019-07-23 18:33:53,143 DEBUG [PEWorker-12] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (39) sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN 2019-07-23 18:33:53,145 INFO [PEWorker-12] procedure.MasterProcedureScheduler(737): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN 2019-07-23 18:33:53,146 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=1 size=0) to run queue because: pid=39, state=WAITING:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor released the exclusive lock 2019-07-23 18:33:53,146 INFO [PEWorker-12] assignment.TransitRegionStateProcedure(188): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN; rit=OFFLINE, location=maven-jdk8-551kz,45127,1563877982046; forceNewPlan=false, retain=false 2019-07-23 18:33:53,147 DEBUG [PEWorker-12] procedure2.RootProcedureState(153): Add procedure pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN as the 4th rollback step 2019-07-23 18:33:53,297 INFO [master/maven-jdk8-551kz:0] balancer.BaseLoadBalancer(1533): Reassigned 1 regions. 1 retained the pre-restart assignment. 2019-07-23 18:33:53,336 DEBUG [master/maven-jdk8-551kz:0] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=1 size=1) to run queue because: pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN has lock 2019-07-23 18:33:53,336 DEBUG [PEWorker-4] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=1 size=0) from run queue because: queue is empty after polling out pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN 2019-07-23 18:33:53,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] master.MasterRpcServices(1253): Checking to see if procedure is done pid=39 2019-07-23 18:33:53,336 INFO [PEWorker-4] assignment.RegionStateStore(199): pid=40 updating hbase:meta row=6c32927df5feade75feea233101e7422, regionState=OPENING, regionLocation=maven-jdk8-551kz,45127,1563877982046 2019-07-23 18:33:53,339 INFO [PEWorker-4] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2019-07-23 18:33:53,339 DEBUG [PEWorker-4] procedure2.RootProcedureState(153): Add procedure pid=40, ppid=39, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_OPENED, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN as the 5th rollback step 2019-07-23 18:33:53,342 DEBUG [PEWorker-4] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=1 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=41, ppid=40, state=RUNNABLE; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure 2019-07-23 18:33:53,342 DEBUG [PEWorker-4] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=1 size=0) from run queue because: queue is empty after polling out pid=41, ppid=40, state=RUNNABLE; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure 2019-07-23 18:33:53,343 DEBUG [PEWorker-4] procedure2.RootProcedureState(153): Add procedure pid=41, ppid=40, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure as the 6th rollback step 2019-07-23 18:33:53,541 INFO [RS_OPEN_REGION-regionserver/maven-jdk8-551kz:0-2] handler.AssignRegionHandler(123): Open testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878032846.6c32927df5feade75feea233101e7422. 2019-07-23 18:33:53,541 DEBUG [RS_OPEN_REGION-regionserver/maven-jdk8-551kz:0-2] regionserver.HRegion(7413): Opening region: {ENCODED => 6c32927df5feade75feea233101e7422, NAME => 'testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878032846.6c32927df5feade75feea233101e7422.', STARTKEY => '', ENDKEY => ''} 2019-07-23 18:33:53,541 DEBUG [RS_OPEN_REGION-regionserver/maven-jdk8-551kz:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testMasterRestartAtRegionSplitPendingCatalogJanitor 6c32927df5feade75feea233101e7422 2019-07-23 18:33:53,542 DEBUG [RS_OPEN_REGION-regionserver/maven-jdk8-551kz:0-2] regionserver.HRegion(868): Instantiated testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878032846.6c32927df5feade75feea233101e7422.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2019-07-23 18:33:53,546 DEBUG [StoreOpener-6c32927df5feade75feea233101e7422-1] util.CommonFSUtils(609): Set storagePolicy=HOT for path=hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/6c32927df5feade75feea233101e7422/info 2019-07-23 18:33:53,546 DEBUG [StoreOpener-6c32927df5feade75feea233101e7422-1] util.CommonFSUtils(609): Set storagePolicy=HOT for path=hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/6c32927df5feade75feea233101e7422/info 2019-07-23 18:33:53,547 INFO [StoreOpener-6c32927df5feade75feea233101e7422-1] hfile.CacheConfig(174): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false for family {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} with blockCache=LruBlockCache{blockCount=0, currentSize=820.31 KB, freeSize=1.07 GB, maxSize=1.07 GB, heapSize=820.31 KB, minSize=1.01 GB, minFactor=0.95, multiSize=518.89 MB, multiFactor=0.5, singleSize=259.45 MB, singleFactor=0.25} 2019-07-23 18:33:53,548 INFO [StoreOpener-6c32927df5feade75feea233101e7422-1] compactions.CompactionConfiguration(151): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory; delete ratio compaction: enable false, threshold 0.450000 2019-07-23 18:33:53,549 INFO [StoreOpener-6c32927df5feade75feea233101e7422-1] regionserver.HStore(332): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2019-07-23 18:33:53,550 DEBUG [RS_OPEN_REGION-regionserver/maven-jdk8-551kz:0-2] regionserver.HRegion(4802): Found 0 recovered edits file(s) under hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/6c32927df5feade75feea233101e7422 2019-07-23 18:33:53,551 DEBUG [RS_OPEN_REGION-regionserver/maven-jdk8-551kz:0-2] regionserver.HRegion(4802): Found 0 recovered edits file(s) under hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/6c32927df5feade75feea233101e7422 2019-07-23 18:33:53,555 DEBUG [RS_OPEN_REGION-regionserver/maven-jdk8-551kz:0-2] regionserver.HRegion(1037): writing seq id for 6c32927df5feade75feea233101e7422 2019-07-23 18:33:53,638 DEBUG [RS_OPEN_REGION-regionserver/maven-jdk8-551kz:0-2] wal.WALSplitUtil(421): Wrote file=hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/6c32927df5feade75feea233101e7422/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2019-07-23 18:33:53,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] master.MasterRpcServices(1253): Checking to see if procedure is done pid=39 2019-07-23 18:33:53,639 INFO [RS_OPEN_REGION-regionserver/maven-jdk8-551kz:0-2] regionserver.HRegion(1054): Opened 6c32927df5feade75feea233101e7422; next sequenceid=2 2019-07-23 18:33:53,641 INFO [RS_OPEN_REGION-regionserver/maven-jdk8-551kz:0-2] regionserver.HRegionServer(2248): Post open deploy tasks for testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878032846.6c32927df5feade75feea233101e7422., openProcId=41, masterSystemTime=1563878033493 2019-07-23 18:33:53,743 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=46015] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=1 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=41, ppid=40, state=RUNNABLE; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure 2019-07-23 18:33:53,744 DEBUG [PEWorker-1] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=1 size=0) from run queue because: queue is empty after polling out pid=41, ppid=40, state=RUNNABLE; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure 2019-07-23 18:33:53,744 WARN [RS-EventLoopGroup-12-2] ipc.AbstractRpcClient(402): Slow ipc call, MethodName=ReportRegionStateTransition, consume time=103, remote address:maven-jdk8-551kz/10.119.14.43:46015 2019-07-23 18:33:53,744 DEBUG [RS_OPEN_REGION-regionserver/maven-jdk8-551kz:0-2] regionserver.HRegionServer(2273): Finished post open deploy task for testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878032846.6c32927df5feade75feea233101e7422. 2019-07-23 18:33:53,745 INFO [RS_OPEN_REGION-regionserver/maven-jdk8-551kz:0-2] handler.AssignRegionHandler(141): Opened testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878032846.6c32927df5feade75feea233101e7422. 2019-07-23 18:33:53,745 INFO [PEWorker-1] assignment.RegionStateStore(199): pid=40 updating hbase:meta row=6c32927df5feade75feea233101e7422, regionState=OPEN, openSeqNum=2, regionLocation=maven-jdk8-551kz,45127,1563877982046 2019-07-23 18:33:53,748 DEBUG [PEWorker-1] procedure2.RootProcedureState(153): Add procedure pid=41, ppid=40, state=SUCCESS, locked=true; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure as the 7th rollback step 2019-07-23 18:33:53,751 DEBUG [PEWorker-1] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=1 size=1) to run queue because: pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN has lock 2019-07-23 18:33:53,751 INFO [PEWorker-1] procedure2.ProcedureExecutor(1820): Finished subprocedure pid=41, resume processing parent pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN 2019-07-23 18:33:53,751 DEBUG [PEWorker-3] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=1 size=0) from run queue because: queue is empty after polling out pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN 2019-07-23 18:33:53,751 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 409msec 2019-07-23 18:33:53,751 DEBUG [PEWorker-3] procedure2.RootProcedureState(153): Add procedure pid=40, ppid=39, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN as the 8th rollback step 2019-07-23 18:33:53,836 DEBUG [PEWorker-3] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=0 size=0) to run queue because: pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN released the shared lock 2019-07-23 18:33:53,838 DEBUG [PEWorker-3] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=39, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:33:53,838 INFO [PEWorker-3] procedure2.ProcedureExecutor(1820): Finished subprocedure pid=40, resume processing parent pid=39, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:33:53,838 DEBUG [PEWorker-6] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=39, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:33:53,838 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN in 610msec 2019-07-23 18:33:53,838 DEBUG [PEWorker-6] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (39) sharedLock=0 size=0) from run queue because: pid=39, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor held the exclusive lock 2019-07-23 18:33:53,840 DEBUG [PEWorker-6] hbase.MetaTableAccessor(2138): Put {"totalColumns":1,"row":"testMasterRestartAtRegionSplitPendingCatalogJanitor","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1563878033840"}]},"ts":"1563878033840"} 2019-07-23 18:33:53,843 INFO [PEWorker-6] hbase.MetaTableAccessor(1673): Updated tableName=testMasterRestartAtRegionSplitPendingCatalogJanitor, state=ENABLED in hbase:meta 2019-07-23 18:33:53,844 DEBUG [PEWorker-6] procedure2.RootProcedureState(153): Add procedure pid=39, state=RUNNABLE:CREATE_TABLE_SYNC_SCHEMA_TO_PEER, locked=true; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 9th rollback step 2019-07-23 18:33:53,845 DEBUG [PEWorker-6] procedure2.RootProcedureState(153): Add procedure pid=39, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 10th rollback step 2019-07-23 18:33:53,847 DEBUG [PEWorker-6] procedure2.RootProcedureState(153): Add procedure pid=39, state=SUCCESS, locked=true; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 11th rollback step 2019-07-23 18:33:53,848 INFO [RS:3;maven-jdk8-551kz:40727] wal.AbstractFSWAL(420): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=maven-jdk8-551kz%2C40727%2C1563878032037, suffix=, logDir=hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/WALs/maven-jdk8-551kz,40727,1563878032037, archiveDir=hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/oldWALs 2019-07-23 18:33:53,849 DEBUG [PEWorker-6] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=0 size=0) to run queue because: pid=39, state=SUCCESS; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor released the exclusive lock 2019-07-23 18:33:53,849 INFO [PEWorker-6] procedure2.ProcedureExecutor(1410): Finished pid=39, state=SUCCESS; CreateTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor in 1.0000sec 2019-07-23 18:33:53,947 DEBUG [RS-EventLoopGroup-14-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(783): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = 127.0.0.1:41521 2019-07-23 18:33:53,947 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(783): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = 127.0.0.1:43977 2019-07-23 18:33:53,951 DEBUG [RS-EventLoopGroup-14-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(783): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = 127.0.0.1:42815 2019-07-23 18:33:53,955 INFO [RS:3;maven-jdk8-551kz:40727] wal.AbstractFSWAL(688): New WAL /user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/WALs/maven-jdk8-551kz,40727,1563878032037/maven-jdk8-551kz%2C40727%2C1563878032037.1563878033851 2019-07-23 18:33:53,955 DEBUG [RS:3;maven-jdk8-551kz:40727] wal.AbstractFSWAL(777): Create new AsyncFSWAL writer with pipeline: [127.0.0.1:41521, 127.0.0.1:43977, 127.0.0.1:42815] 2019-07-23 18:33:54,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] master.MasterRpcServices(1253): Checking to see if procedure is done pid=39 2019-07-23 18:33:54,140 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3608): Operation: CREATE, Table Name: default:testMasterRestartAtRegionSplitPendingCatalogJanitor, procId: 39 completed 2019-07-23 18:33:54,141 DEBUG [Time-limited test] hbase.HBaseTestingUtility(3564): Waiting until all regions of table testMasterRestartAtRegionSplitPendingCatalogJanitor get assigned. Timeout = 60000ms 2019-07-23 18:33:54,141 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2019-07-23 18:33:54,147 INFO [Time-limited test] hbase.HBaseTestingUtility(3616): All regions for table testMasterRestartAtRegionSplitPendingCatalogJanitor assigned to meta. Checking AM states. 2019-07-23 18:33:54,148 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2019-07-23 18:33:54,148 INFO [Time-limited test] hbase.HBaseTestingUtility(3636): All regions for table testMasterRestartAtRegionSplitPendingCatalogJanitor assigned. 2019-07-23 18:33:54,151 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [10,000] milli-secs(wait.for.ratio=[1]) 2019-07-23 18:33:54,151 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2019-07-23 18:33:54,151 INFO [Time-limited test] regionserver.TestSplitTr ...[truncated 159274 chars]... IGN 2019-07-23 18:34:02,945 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 603msec 2019-07-23 18:34:02,945 DEBUG [PEWorker-7] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=1 size=0) from run queue because: queue is empty after polling out pid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN 2019-07-23 18:34:02,945 DEBUG [PEWorker-7] procedure2.RootProcedureState(153): Add procedure pid=49, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN as the 4th rollback step 2019-07-23 18:34:02,947 DEBUG [PEWorker-7] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=0 size=0) to run queue because: pid=49, state=SUCCESS; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN released the shared lock 2019-07-23 18:34:02,947 INFO [PEWorker-7] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=6c32927df5feade75feea233101e7422, ASSIGN in 999msec 2019-07-23 18:34:03,035 INFO [Time-limited test] throttle.PressureAwareThroughputController(153): 4c0e59f758ca77fd0ac07664b82c7847#info#compaction#9 average throughput is 0.67 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2019-07-23 18:34:03,038 INFO [IPC Server handler 2 on 46863] blockmanagement.BlockManager(2599): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43977 is added to blk_1073741865_1041{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction, ReplicaUnderConstruction, ReplicaUnderConstruction]} size 0 2019-07-23 18:34:03,039 INFO [IPC Server handler 6 on 46863] blockmanagement.BlockManager(2599): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41521 is added to blk_1073741865_1041{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction, ReplicaUnderConstruction, ReplicaUnderConstruction]} size 0 2019-07-23 18:34:03,039 INFO [IPC Server handler 9 on 46863] blockmanagement.BlockManager(2599): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42815 is added to blk_1073741865_1041{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction, ReplicaUnderConstruction, ReplicaUnderConstruction]} size 0 2019-07-23 18:34:03,046 DEBUG [Time-limited test] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847/.tmp/info/68f0bcd0c23947f3bd4b8242f250f516 as hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847/info/68f0bcd0c23947f3bd4b8242f250f516 2019-07-23 18:34:03,140 INFO [Time-limited test] regionserver.HStore(1555): Completed major compaction of 1 (all) file(s) in info of 4c0e59f758ca77fd0ac07664b82c7847 into 68f0bcd0c23947f3bd4b8242f250f516(size=285.7 K), total size for store is 285.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2019-07-23 18:34:03,140 DEBUG [Time-limited test] regionserver.HRegion(2342): Compaction status journal: Compacting store info at 1563878042536 Compaction complete at 1563878043140 2019-07-23 18:34:03,144 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=45127] regionserver.HRegion(1505): Region testMasterRestartAtRegionSplitPendingCatalogJanitor,nwf,1563878037242.4c0e59f758ca77fd0ac07664b82c7847. is not mergeable because it has references 2019-07-23 18:34:03,235 DEBUG [Time-limited test] regionserver.HStore(2617): Moving the files [hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847/info/2487b940993e46feb2e7d7c16ed9729c.6c32927df5feade75feea233101e7422-hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/6c32927df5feade75feea233101e7422/info/2487b940993e46feb2e7d7c16ed9729c-top, hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847/info/6c4a896489034e99ab87cd681bc5deb1] to archive 2019-07-23 18:34:03,237 DEBUG [Time-limited test] backup.HFileArchiver(323): Archiving compacted files. 2019-07-23 18:34:03,239 DEBUG [Time-limited test] backup.HFileArchiver(512): Archived from FileableStoreFile, hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847/info/2487b940993e46feb2e7d7c16ed9729c.6c32927df5feade75feea233101e7422 to hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/archive/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847/info/2487b940993e46feb2e7d7c16ed9729c.6c32927df5feade75feea233101e7422 2019-07-23 18:34:03,240 DEBUG [Time-limited test] backup.HFileArchiver(512): Archived from FileableStoreFile, hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847/info/6c4a896489034e99ab87cd681bc5deb1 to hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/archive/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847/info/6c4a896489034e99ab87cd681bc5deb1 2019-07-23 18:34:03,241 INFO [Time-limited test] hbase.ScheduledChore(180): Chore: CompactedHFilesCleaner was stopped 2019-07-23 18:34:03,241 INFO [Time-limited test] regionserver.TestSplitTransactionOnCluster(572): Starting run of CatalogJanitor 2019-07-23 18:34:03,241 WARN [Time-limited test] master.CatalogJanitor(119): CatalogJanitor is disabled! Enabled=true, maintenanceMode=false, am=org.apache.hadoop.hbase.master.assignment.AssignmentManager@1ce25bee, metaLoaded=true, hasRIT=true clusterShutDown=false 2019-07-23 18:34:03,248 DEBUG [Default-IPC-NioEventLoopGroup-13-3] ipc.FailedServers(56): Added failed server with address maven-jdk8-551kz/10.119.14.43:46015 to list caused by com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: maven-jdk8-551kz/10.119.14.43:46015 2019-07-23 18:34:03,248 WARN [Time-limited test] client.ConnectionImplementation(1892): Checking master connection java.net.ConnectException: Call to maven-jdk8-551kz/10.119.14.43:46015 failed on connection exception: com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: maven-jdk8-551kz/10.119.14.43:46015 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:178) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:414) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:97) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:445) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:441) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:326) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:312) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:304) at com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1426) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:326) at com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:312) at com.xiaomi.infra.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:924) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:183) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:271) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:265) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:502) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:495) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:474) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:415) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:540) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:533) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:114) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:665) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) at com.xiaomi.infra.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) Caused by: com.xiaomi.infra.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: maven-jdk8-551kz/10.119.14.43:46015 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at com.xiaomi.infra.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) at com.xiaomi.infra.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ... 7 more Caused by: java.net.ConnectException: Connection refused ... 11 more 2019-07-23 18:34:03,342 INFO [RS-EventLoopGroup-15-3] ipc.ServerRpcConnection(556): Connection from 10.119.14.43:48378, version=2.2.0-mdh3.1.1, sasl=false, ugi=default (auth:SIMPLE), service=MasterService 2019-07-23 18:34:03,345 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(484): master:35991-0x16c1e642279000c, quorum=localhost:62549, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2019-07-23 18:34:03,347 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35991] master.MasterRpcServices(461): Client=default//10.119.14.43 set balanceSwitch=true 2019-07-23 18:34:03,438 INFO [Time-limited test] regionserver.TestSplitTransactionOnCluster(161): Tear down, remove table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:03,439 INFO [Time-limited test] client.HBaseAdmin$15(917): Started disable of testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:03,439 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35991] master.HMaster$11(2360): Client=default//10.119.14.43 disable testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:03,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35991] procedure2.ProcedureExecutor(1027): Stored pid=51, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:03,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35991] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=51, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:03,544 DEBUG [PEWorker-16] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=51, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:03,544 DEBUG [PEWorker-16] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=0 size=0) from run queue because: pid=51, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor held the exclusive lock 2019-07-23 18:34:03,547 DEBUG [PEWorker-16] procedure2.RootProcedureState(153): Add procedure pid=51, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 0th rollback step 2019-07-23 18:34:03,548 WARN [Default-IPC-NioEventLoopGroup-13-4] ipc.AbstractRpcClient(402): Slow ipc call, MethodName=DisableTable, consume time=109, remote address:maven-jdk8-551kz/10.119.14.43:35991 2019-07-23 18:34:03,548 DEBUG [PEWorker-16] procedure2.RootProcedureState(153): Add procedure pid=51, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE, locked=true; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 1th rollback step 2019-07-23 18:34:03,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35991] master.MasterRpcServices(1253): Checking to see if procedure is done pid=51 2019-07-23 18:34:03,550 DEBUG [PEWorker-16] hbase.MetaTableAccessor(2138): Put {"totalColumns":1,"row":"testMasterRestartAtRegionSplitPendingCatalogJanitor","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1563878043549"}]},"ts":"1563878043549"} 2019-07-23 18:34:03,635 INFO [PEWorker-16] hbase.MetaTableAccessor(1673): Updated tableName=testMasterRestartAtRegionSplitPendingCatalogJanitor, state=DISABLING in hbase:meta 2019-07-23 18:34:03,636 INFO [PEWorker-16] procedure.DisableTableProcedure(295): Set testMasterRestartAtRegionSplitPendingCatalogJanitor to state=DISABLING 2019-07-23 18:34:03,636 DEBUG [PEWorker-16] procedure2.RootProcedureState(153): Add procedure pid=51, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE, locked=true; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 2th rollback step 2019-07-23 18:34:03,637 INFO [PEWorker-16] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=d80c3718d070570da627cb6a1a9cac70, UNASSIGN}, {pid=53, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=4c0e59f758ca77fd0ac07664b82c7847, UNASSIGN}] 2019-07-23 18:34:03,637 DEBUG [PEWorker-16] procedure2.RootProcedureState(153): Add procedure pid=51, state=WAITING:DISABLE_TABLE_ADD_REPLICATION_BARRIER, locked=true; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 3th rollback step 2019-07-23 18:34:03,639 DEBUG [PEWorker-16] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=0 size=1) to run queue because: pid=52, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=d80c3718d070570da627cb6a1a9cac70, UNASSIGN has the excusive lock access 2019-07-23 18:34:03,639 DEBUG [PEWorker-16] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=0 size=2) to run queue because: pid=53, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=4c0e59f758ca77fd0ac07664b82c7847, UNASSIGN has the excusive lock access 2019-07-23 18:34:03,639 DEBUG [PEWorker-16] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=52, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=d80c3718d070570da627cb6a1a9cac70, UNASSIGN 2019-07-23 18:34:03,641 INFO [PEWorker-9] procedure.MasterProcedureScheduler(737): Took xlock for pid=53, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=4c0e59f758ca77fd0ac07664b82c7847, UNASSIGN 2019-07-23 18:34:03,641 INFO [PEWorker-16] procedure.MasterProcedureScheduler(737): Took xlock for pid=52, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=d80c3718d070570da627cb6a1a9cac70, UNASSIGN 2019-07-23 18:34:03,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35991] master.MasterRpcServices(1253): Checking to see if procedure is done pid=51 2019-07-23 18:34:03,742 INFO [PEWorker-9] assignment.RegionStateStore(199): pid=53 updating hbase:meta row=4c0e59f758ca77fd0ac07664b82c7847, regionState=CLOSING, regionLocation=maven-jdk8-551kz,45127,1563877982046 2019-07-23 18:34:03,745 INFO [PEWorker-9] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=54, ppid=53, state=RUNNABLE; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2019-07-23 18:34:03,745 DEBUG [PEWorker-9] procedure2.RootProcedureState(153): Add procedure pid=53, ppid=51, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=4c0e59f758ca77fd0ac07664b82c7847, UNASSIGN as the 4th rollback step 2019-07-23 18:34:03,746 DEBUG [PEWorker-9] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=2 size=1) to run queue because: pid=54, ppid=53, state=RUNNABLE; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure has the excusive lock access 2019-07-23 18:34:03,746 INFO [PEWorker-16] assignment.RegionStateStore(199): pid=52 updating hbase:meta row=d80c3718d070570da627cb6a1a9cac70, regionState=CLOSING, regionLocation=maven-jdk8-551kz,45127,1563877982046 2019-07-23 18:34:03,746 DEBUG [PEWorker-11] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=2 size=0) from run queue because: queue is empty after polling out pid=54, ppid=53, state=RUNNABLE; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2019-07-23 18:34:03,749 INFO [PEWorker-16] procedure2.ProcedureExecutor(1680): Initialized subprocedures=[{pid=55, ppid=52, state=RUNNABLE; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure}] 2019-07-23 18:34:03,749 DEBUG [PEWorker-16] procedure2.RootProcedureState(153): Add procedure pid=52, ppid=51, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=d80c3718d070570da627cb6a1a9cac70, UNASSIGN as the 5th rollback step 2019-07-23 18:34:03,835 DEBUG [PEWorker-16] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=2 size=1) to run queue because: pid=55, ppid=52, state=RUNNABLE; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure has the excusive lock access 2019-07-23 18:34:03,835 DEBUG [PEWorker-11] procedure2.RootProcedureState(153): Add procedure pid=54, ppid=53, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure as the 6th rollback step 2019-07-23 18:34:03,835 DEBUG [PEWorker-13] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=2 size=0) from run queue because: queue is empty after polling out pid=55, ppid=52, state=RUNNABLE; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2019-07-23 18:34:03,837 DEBUG [PEWorker-13] procedure2.RootProcedureState(153): Add procedure pid=55, ppid=52, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure as the 7th rollback step 2019-07-23 18:34:03,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35991] master.MasterRpcServices(1253): Checking to see if procedure is done pid=51 2019-07-23 18:34:04,036 INFO [RS_CLOSE_REGION-regionserver/maven-jdk8-551kz:0-0] handler.UnassignRegionHandler(106): Close d80c3718d070570da627cb6a1a9cac70 2019-07-23 18:34:04,036 INFO [RS_CLOSE_REGION-regionserver/maven-jdk8-551kz:0-1] handler.UnassignRegionHandler(106): Close 4c0e59f758ca77fd0ac07664b82c7847 2019-07-23 18:34:04,036 DEBUG [RS_CLOSE_REGION-regionserver/maven-jdk8-551kz:0-0] regionserver.HRegion(1657): Closing d80c3718d070570da627cb6a1a9cac70, disabling compactions & flushes 2019-07-23 18:34:04,036 DEBUG [RS_CLOSE_REGION-regionserver/maven-jdk8-551kz:0-1] regionserver.HRegion(1657): Closing 4c0e59f758ca77fd0ac07664b82c7847, disabling compactions & flushes 2019-07-23 18:34:04,036 DEBUG [RS_CLOSE_REGION-regionserver/maven-jdk8-551kz:0-0] regionserver.HRegion(1697): Updates disabled for region testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878037242.d80c3718d070570da627cb6a1a9cac70. 2019-07-23 18:34:04,036 DEBUG [RS_CLOSE_REGION-regionserver/maven-jdk8-551kz:0-1] regionserver.HRegion(1697): Updates disabled for region testMasterRestartAtRegionSplitPendingCatalogJanitor,nwf,1563878037242.4c0e59f758ca77fd0ac07664b82c7847. 2019-07-23 18:34:04,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35991] master.MasterRpcServices(1253): Checking to see if procedure is done pid=51 2019-07-23 18:34:04,540 DEBUG [RS_CLOSE_REGION-regionserver/maven-jdk8-551kz:0-1] wal.WALSplitUtil(421): Wrote file=hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=8 2019-07-23 18:34:04,540 DEBUG [RS_CLOSE_REGION-regionserver/maven-jdk8-551kz:0-0] wal.WALSplitUtil(421): Wrote file=hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/d80c3718d070570da627cb6a1a9cac70/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=8 2019-07-23 18:34:04,542 INFO [RS_CLOSE_REGION-regionserver/maven-jdk8-551kz:0-0] regionserver.HRegion(1814): Closed testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878037242.d80c3718d070570da627cb6a1a9cac70. 2019-07-23 18:34:04,543 INFO [RS_CLOSE_REGION-regionserver/maven-jdk8-551kz:0-1] regionserver.HRegion(1814): Closed testMasterRestartAtRegionSplitPendingCatalogJanitor,nwf,1563878037242.4c0e59f758ca77fd0ac07664b82c7847. 2019-07-23 18:34:04,647 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=35991] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=2 size=1) to run queue because: pid=55, ppid=52, state=RUNNABLE; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure has the excusive lock access 2019-07-23 18:34:04,648 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=35991] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=2 size=2) to run queue because: pid=54, ppid=53, state=RUNNABLE; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure has the excusive lock access 2019-07-23 18:34:04,648 WARN [RS-EventLoopGroup-12-1] ipc.AbstractRpcClient(402): Slow ipc call, MethodName=ReportRegionStateTransition, consume time=105, remote address:maven-jdk8-551kz/10.119.14.43:35991 2019-07-23 18:34:04,648 DEBUG [PEWorker-14] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=2 size=0) from run queue because: queue is empty after polling out pid=55, ppid=52, state=RUNNABLE; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure 2019-07-23 18:34:04,648 WARN [RS-EventLoopGroup-12-1] ipc.AbstractRpcClient(402): Slow ipc call, MethodName=ReportRegionStateTransition, consume time=105, remote address:maven-jdk8-551kz/10.119.14.43:35991 2019-07-23 18:34:04,648 INFO [RS_CLOSE_REGION-regionserver/maven-jdk8-551kz:0-0] handler.UnassignRegionHandler(130): Closed d80c3718d070570da627cb6a1a9cac70 2019-07-23 18:34:04,648 INFO [RS_CLOSE_REGION-regionserver/maven-jdk8-551kz:0-1] handler.UnassignRegionHandler(130): Closed 4c0e59f758ca77fd0ac07664b82c7847 2019-07-23 18:34:04,649 INFO [PEWorker-12] assignment.RegionStateStore(199): pid=53 updating hbase:meta row=4c0e59f758ca77fd0ac07664b82c7847, regionState=CLOSED 2019-07-23 18:34:04,735 DEBUG [PEWorker-12] procedure2.RootProcedureState(153): Add procedure pid=54, ppid=53, state=SUCCESS, locked=true; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure as the 8th rollback step 2019-07-23 18:34:04,736 INFO [PEWorker-14] assignment.RegionStateStore(199): pid=52 updating hbase:meta row=d80c3718d070570da627cb6a1a9cac70, regionState=CLOSED 2019-07-23 18:34:04,740 DEBUG [PEWorker-14] procedure2.RootProcedureState(153): Add procedure pid=55, ppid=52, state=SUCCESS, locked=true; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure as the 9th rollback step 2019-07-23 18:34:04,742 DEBUG [PEWorker-12] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=2 size=1) to run queue because: pid=53, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=4c0e59f758ca77fd0ac07664b82c7847, UNASSIGN has lock 2019-07-23 18:34:04,742 INFO [PEWorker-12] procedure2.ProcedureExecutor(1820): Finished subprocedure pid=54, resume processing parent pid=53, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=4c0e59f758ca77fd0ac07664b82c7847, UNASSIGN 2019-07-23 18:34:04,742 INFO [PEWorker-12] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=53, state=SUCCESS; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure in 990msec 2019-07-23 18:34:04,742 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=2 size=0) from run queue because: queue is empty after polling out pid=53, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=4c0e59f758ca77fd0ac07664b82c7847, UNASSIGN 2019-07-23 18:34:04,743 DEBUG [PEWorker-15] procedure2.RootProcedureState(153): Add procedure pid=53, ppid=51, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=4c0e59f758ca77fd0ac07664b82c7847, UNASSIGN as the 10th rollback step 2019-07-23 18:34:04,744 DEBUG [PEWorker-14] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=2 size=1) to run queue because: pid=52, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=d80c3718d070570da627cb6a1a9cac70, UNASSIGN has lock 2019-07-23 18:34:04,744 INFO [PEWorker-14] procedure2.ProcedureExecutor(1820): Finished subprocedure pid=55, resume processing parent pid=52, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=d80c3718d070570da627cb6a1a9cac70, UNASSIGN 2019-07-23 18:34:04,744 DEBUG [PEWorker-2] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=2 size=0) from run queue because: queue is empty after polling out pid=52, ppid=51, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=d80c3718d070570da627cb6a1a9cac70, UNASSIGN 2019-07-23 18:34:04,744 INFO [PEWorker-14] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=52, state=SUCCESS; org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure in 991msec 2019-07-23 18:34:04,744 DEBUG [PEWorker-2] procedure2.RootProcedureState(153): Add procedure pid=52, ppid=51, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=d80c3718d070570da627cb6a1a9cac70, UNASSIGN as the 11th rollback step 2019-07-23 18:34:04,745 INFO [PEWorker-15] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=51, state=SUCCESS; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=4c0e59f758ca77fd0ac07664b82c7847, UNASSIGN in 1.1060sec 2019-07-23 18:34:04,848 DEBUG [PEWorker-2] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=0 size=1) to run queue because: pid=51, state=RUNNABLE:DISABLE_TABLE_ADD_REPLICATION_BARRIER, locked=true; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor has lock 2019-07-23 18:34:04,848 INFO [PEWorker-2] procedure2.ProcedureExecutor(1820): Finished subprocedure pid=52, resume processing parent pid=51, state=RUNNABLE:DISABLE_TABLE_ADD_REPLICATION_BARRIER, locked=true; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:04,848 DEBUG [PEWorker-6] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (51) sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=51, state=RUNNABLE:DISABLE_TABLE_ADD_REPLICATION_BARRIER, locked=true; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:04,848 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; TransitRegionStateProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor, region=d80c3718d070570da627cb6a1a9cac70, UNASSIGN in 1.1070sec 2019-07-23 18:34:04,848 DEBUG [PEWorker-6] procedure2.RootProcedureState(153): Add procedure pid=51, state=RUNNABLE:DISABLE_TABLE_SET_DISABLED_TABLE_STATE, locked=true; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 12th rollback step 2019-07-23 18:34:04,850 DEBUG [PEWorker-6] hbase.MetaTableAccessor(2138): Put {"totalColumns":1,"row":"testMasterRestartAtRegionSplitPendingCatalogJanitor","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1563878044849"}]},"ts":"1563878044849"} 2019-07-23 18:34:04,935 INFO [PEWorker-6] hbase.MetaTableAccessor(1673): Updated tableName=testMasterRestartAtRegionSplitPendingCatalogJanitor, state=DISABLED in hbase:meta 2019-07-23 18:34:04,937 INFO [PEWorker-6] procedure.DisableTableProcedure(307): Set testMasterRestartAtRegionSplitPendingCatalogJanitor to state=DISABLED 2019-07-23 18:34:04,937 DEBUG [PEWorker-6] procedure2.RootProcedureState(153): Add procedure pid=51, state=RUNNABLE:DISABLE_TABLE_POST_OPERATION, locked=true; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 13th rollback step 2019-07-23 18:34:04,938 DEBUG [PEWorker-6] procedure2.RootProcedureState(153): Add procedure pid=51, state=SUCCESS, locked=true; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 14th rollback step 2019-07-23 18:34:04,940 DEBUG [PEWorker-6] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=0 size=0) to run queue because: pid=51, state=SUCCESS; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor released the exclusive lock 2019-07-23 18:34:04,940 INFO [PEWorker-6] procedure2.ProcedureExecutor(1410): Finished pid=51, state=SUCCESS; DisableTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor in 1.4970sec 2019-07-23 18:34:05,036 WARN [HBase-Metrics2-1] impl.MetricsConfig(124): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2019-07-23 18:34:05,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35991] master.MasterRpcServices(1253): Checking to see if procedure is done pid=51 2019-07-23 18:34:05,038 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3608): Operation: DISABLE, Table Name: default:testMasterRestartAtRegionSplitPendingCatalogJanitor, procId: 51 completed 2019-07-23 18:34:05,040 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35991] master.HMaster$5(2170): Client=default//10.119.14.43 delete testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:05,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35991] procedure2.ProcedureExecutor(1027): Stored pid=56, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:05,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35991] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=56, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:05,147 DEBUG [PEWorker-4] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=56, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:05,148 DEBUG [PEWorker-4] procedure.MasterProcedureScheduler(362): Remove TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=true (56) sharedLock=0 size=0) from run queue because: pid=56, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor held the exclusive lock 2019-07-23 18:34:05,151 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(99): Waiting for RIT for pid=56, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:05,151 DEBUG [PEWorker-4] procedure2.RootProcedureState(153): Add procedure pid=56, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 0th rollback step 2019-07-23 18:34:05,151 WARN [Default-IPC-NioEventLoopGroup-13-4] ipc.AbstractRpcClient(402): Slow ipc call, MethodName=DeleteTable, consume time=112, remote address:maven-jdk8-551kz/10.119.14.43:35991 2019-07-23 18:34:05,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35991] master.MasterRpcServices(1253): Checking to see if procedure is done pid=56 2019-07-23 18:34:05,235 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(110): Deleting regions from META for pid=56, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:05,235 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2138): Delete {"totalColumns":1,"row":"testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878037242.d80c3718d070570da627cb6a1a9cac70.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1563878045235"}]},"ts":"9223372036854775807"} 2019-07-23 18:34:05,236 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2138): Delete {"totalColumns":1,"row":"testMasterRestartAtRegionSplitPendingCatalogJanitor,nwf,1563878037242.4c0e59f758ca77fd0ac07664b82c7847.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1563878045235"}]},"ts":"9223372036854775807"} 2019-07-23 18:34:05,239 INFO [PEWorker-4] hbase.MetaTableAccessor(1826): Deleted 2 regions from META 2019-07-23 18:34:05,239 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1827): Deleted regions: [{ENCODED => d80c3718d070570da627cb6a1a9cac70, NAME => 'testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878037242.d80c3718d070570da627cb6a1a9cac70.', STARTKEY => '', ENDKEY => 'nwf'}, {ENCODED => 4c0e59f758ca77fd0ac07664b82c7847, NAME => 'testMasterRestartAtRegionSplitPendingCatalogJanitor,nwf,1563878037242.4c0e59f758ca77fd0ac07664b82c7847.', STARTKEY => 'nwf', ENDKEY => ''}] 2019-07-23 18:34:05,242 WARN [PEWorker-4] procedure.DeleteTableProcedure(376): Deleting some vestigial 1 rows of testMasterRestartAtRegionSplitPendingCatalogJanitor from hbase:meta 2019-07-23 18:34:05,245 DEBUG [PEWorker-4] procedure2.RootProcedureState(153): Add procedure pid=56, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 1th rollback step 2019-07-23 18:34:05,246 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(115): Deleting regions from filesystem for pid=56, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:05,335 DEBUG [HFileArchiver-6] backup.HFileArchiver(129): ARCHIVING hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/d80c3718d070570da627cb6a1a9cac70 2019-07-23 18:34:05,335 DEBUG [HFileArchiver-7] backup.HFileArchiver(129): ARCHIVING hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847 2019-07-23 18:34:05,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35991] master.MasterRpcServices(1253): Checking to see if procedure is done pid=56 2019-07-23 18:34:05,338 DEBUG [HFileArchiver-7] backup.HFileArchiver(158): Archiving [FileablePath, hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847/info, FileablePath, hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847/recovered.edits] 2019-07-23 18:34:05,338 DEBUG [HFileArchiver-6] backup.HFileArchiver(158): Archiving [FileablePath, hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/d80c3718d070570da627cb6a1a9cac70/info, FileablePath, hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/d80c3718d070570da627cb6a1a9cac70/recovered.edits] 2019-07-23 18:34:05,341 DEBUG [HFileArchiver-7] backup.HFileArchiver(512): Archived from FileablePath, hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847/info/68f0bcd0c23947f3bd4b8242f250f516 to hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/archive/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847/info/68f0bcd0c23947f3bd4b8242f250f516 2019-07-23 18:34:05,341 DEBUG [HFileArchiver-6] backup.HFileArchiver(512): Archived from FileablePath, hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/d80c3718d070570da627cb6a1a9cac70/info/8dc8032ff29a44079a32ec3484eec325 to hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/archive/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/d80c3718d070570da627cb6a1a9cac70/info/8dc8032ff29a44079a32ec3484eec325 2019-07-23 18:34:05,345 DEBUG [HFileArchiver-7] backup.HFileArchiver(512): Archived from FileablePath, hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847/recovered.edits/13.seqid to hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/archive/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847/recovered.edits/13.seqid 2019-07-23 18:34:05,345 DEBUG [HFileArchiver-6] backup.HFileArchiver(512): Archived from FileablePath, hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/d80c3718d070570da627cb6a1a9cac70/recovered.edits/13.seqid to hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/archive/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/d80c3718d070570da627cb6a1a9cac70/recovered.edits/13.seqid 2019-07-23 18:34:05,346 DEBUG [HFileArchiver-7] backup.HFileArchiver(526): Deleted hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/4c0e59f758ca77fd0ac07664b82c7847 2019-07-23 18:34:05,347 DEBUG [HFileArchiver-6] backup.HFileArchiver(526): Deleted hdfs://localhost:46863/user/default/test-data/4aa48a8d-3a95-9a8d-ec29-57a6e17b297f/.tmp/data/default/testMasterRestartAtRegionSplitPendingCatalogJanitor/d80c3718d070570da627cb6a1a9cac70 2019-07-23 18:34:05,347 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(327): Table 'testMasterRestartAtRegionSplitPendingCatalogJanitor' archived! 2019-07-23 18:34:05,434 DEBUG [PEWorker-4] procedure2.RootProcedureState(153): Add procedure pid=56, state=RUNNABLE:DELETE_TABLE_UPDATE_DESC_CACHE, locked=true; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 2th rollback step 2019-07-23 18:34:05,436 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(121): Deleting descriptor for pid=56, state=RUNNABLE:DELETE_TABLE_UPDATE_DESC_CACHE, locked=true; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:05,436 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(413): Removing 'testMasterRestartAtRegionSplitPendingCatalogJanitor' descriptor. 2019-07-23 18:34:05,437 DEBUG [PEWorker-4] procedure2.RootProcedureState(153): Add procedure pid=56, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 3th rollback step 2019-07-23 18:34:05,438 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(126): Deleting assignment state for pid=56, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:05,438 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(403): Removing 'testMasterRestartAtRegionSplitPendingCatalogJanitor' from region states. 2019-07-23 18:34:05,438 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2138): Delete {"totalColumns":1,"row":"testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878032846.6c32927df5feade75feea233101e7422.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1563878045438"}]},"ts":"9223372036854775807"} 2019-07-23 18:34:05,439 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2138): Delete {"totalColumns":1,"row":"testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878037242.d80c3718d070570da627cb6a1a9cac70.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1563878045438"}]},"ts":"9223372036854775807"} 2019-07-23 18:34:05,439 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2138): Delete {"totalColumns":1,"row":"testMasterRestartAtRegionSplitPendingCatalogJanitor,nwf,1563878037242.4c0e59f758ca77fd0ac07664b82c7847.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1563878045438"}]},"ts":"9223372036854775807"} 2019-07-23 18:34:05,442 INFO [PEWorker-4] hbase.MetaTableAccessor(1826): Deleted 3 regions from META 2019-07-23 18:34:05,442 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1827): Deleted regions: [{ENCODED => 6c32927df5feade75feea233101e7422, NAME => 'testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878032846.6c32927df5feade75feea233101e7422.', STARTKEY => '', ENDKEY => '', OFFLINE => true, SPLIT => true}, {ENCODED => d80c3718d070570da627cb6a1a9cac70, NAME => 'testMasterRestartAtRegionSplitPendingCatalogJanitor,,1563878037242.d80c3718d070570da627cb6a1a9cac70.', STARTKEY => '', ENDKEY => 'nwf'}, {ENCODED => 4c0e59f758ca77fd0ac07664b82c7847, NAME => 'testMasterRestartAtRegionSplitPendingCatalogJanitor,nwf,1563878037242.4c0e59f758ca77fd0ac07664b82c7847.', STARTKEY => 'nwf', ENDKEY => ''}] 2019-07-23 18:34:05,442 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(407): Marking 'testMasterRestartAtRegionSplitPendingCatalogJanitor' as deleted. 2019-07-23 18:34:05,443 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2138): Delete {"totalColumns":1,"row":"testMasterRestartAtRegionSplitPendingCatalogJanitor","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1563878045442"}]},"ts":"9223372036854775807"} 2019-07-23 18:34:05,445 INFO [PEWorker-4] hbase.MetaTableAccessor(1697): Deleted table testMasterRestartAtRegionSplitPendingCatalogJanitor state from META 2019-07-23 18:34:05,446 DEBUG [PEWorker-4] procedure2.RootProcedureState(153): Add procedure pid=56, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 4th rollback step 2019-07-23 18:34:05,447 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(132): Finished pid=56, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor 2019-07-23 18:34:05,447 DEBUG [PEWorker-4] procedure2.RootProcedureState(153): Add procedure pid=56, state=SUCCESS, locked=true; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor as the 5th rollback step 2019-07-23 18:34:05,451 DEBUG [PEWorker-4] procedure.MasterProcedureScheduler(352): Add TableQueue(testMasterRestartAtRegionSplitPendingCatalogJanitor, xlock=false sharedLock=0 size=0) to run queue because: pid=56, state=SUCCESS; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor released the exclusive lock 2019-07-23 18:34:05,451 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, state=SUCCESS; DeleteTableProcedure table=testMasterRestartAtRegionSplitPendingCatalogJanitor in 407msec 2019-07-23 18:34:05,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35991] master.MasterRpcServices(1253): Checking to see if procedure is done pid=56 2019-07-23 18:34:05,538 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3608): Operation: DELETE, Table Name: default:testMasterRestartAtRegionSplitPendingCatalogJanitor, procId: 56 completed Page generated: 2019-7-25 ??11?04?33?REST APIJenkins ver. 2.150.2