2020-05-31 19:24:59,046 DEBUG [main] ipc.RpcConnection(126): Use SIMPLE authentication for service MasterService, sasl=false 2020-05-31 19:24:59,046 DEBUG [main] ipc.BlockingRpcConnection(431): Connecting to foo.com/10.55.1.229:56773 2020-05-31 19:24:59,047 DEBUG [RpcServer.listener,port=56773] ipc.RpcServer$Listener(950): RpcServer.listener,port=56773: connection from 10.55.1.229:56798; # active connections: 2 2020-05-31 19:24:59,048 INFO [RpcServer.reader=2,bindAddress=foo.com,port=56773] ipc.RpcServer$Connection(1857): Auth successful for foo (auth:SIMPLE) 2020-05-31 19:24:59,048 INFO [RpcServer.reader=2,bindAddress=foo.com,port=56773] ipc.RpcServer$Connection(1887): Connection from 10.55.1.229 port: 56798 with version info: version: "1.7.0-SNAPSHOT" url: "git://foo.com/Users/foo/IdeaProjects/hbase-master" revision: "90b66544f0841f03bd0096d0679f526355a16fcd" user: "foo" date: "Sun May 31 14:23:50 PDT 2020" src_checksum: "9b20e6bd764436f9d419120b9f623653" version_major: 1 version_minor: 7 2020-05-31 19:24:59,055 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] master.HMaster$1(1831): Client=foo//10.55.1.229 create 'testFIFOCompactionPolicyExpiredEmptyHFiles', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hstore.defaultengine.compactionpolicy.class' => 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy', 'hbase.regionserver.region.split.policy' => 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy'}}, {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => '1 SECOND', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2020-05-31 19:24:59,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] procedure2.ProcedureExecutor(769): Procedure CreateTableProcedure (table=testFIFOCompactionPolicyExpiredEmptyHFiles) id=4 owner=foo state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2020-05-31 19:24:59,166 DEBUG [ProcedureExecutor-3] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testFIFOCompactionPolicyExpiredEmptyHFiles/write-master:567730000000000 2020-05-31 19:24:59,169 DEBUG [ProcedureExecutor-3] zookeeper.ZKUtil(677): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Unable to get data of znode /hbase/table/testFIFOCompactionPolicyExpiredEmptyHFiles because node does not exist (not an error) 2020-05-31 19:24:59,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] master.MasterRpcServices(1078): Checking to see if procedure is done procId=4 2020-05-31 19:24:59,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] master.MasterRpcServices(1078): Checking to see if procedure is done procId=4 2020-05-31 19:24:59,289 DEBUG [ProcedureExecutor-3] util.FSTableDescriptors(660): Wrote descriptor into: hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/.tmp/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/.tabledesc/.tableinfo.0000000001 2020-05-31 19:24:59,290 INFO [RegionOpenAndInitThread-testFIFOCompactionPolicyExpiredEmptyHFiles-1] regionserver.HRegion(6887): creating HRegion testFIFOCompactionPolicyExpiredEmptyHFiles HTD == 'testFIFOCompactionPolicyExpiredEmptyHFiles', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hstore.defaultengine.compactionpolicy.class' => 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy', 'hbase.regionserver.region.split.policy' => 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy'}}, {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => '1 SECOND', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/.tmp Table name == testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:24:59,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] master.MasterRpcServices(1078): Checking to see if procedure is done procId=4 2020-05-31 19:24:59,702 DEBUG [RegionOpenAndInitThread-testFIFOCompactionPolicyExpiredEmptyHFiles-1] regionserver.HRegion(832): Instantiated testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:24:59,703 DEBUG [RegionOpenAndInitThread-testFIFOCompactionPolicyExpiredEmptyHFiles-1] regionserver.HRegion(1544): Closing testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f.: disabling compactions & flushes 2020-05-31 19:24:59,703 DEBUG [RegionOpenAndInitThread-testFIFOCompactionPolicyExpiredEmptyHFiles-1] regionserver.HRegion(1571): Updates disabled for region testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:24:59,703 INFO [RegionOpenAndInitThread-testFIFOCompactionPolicyExpiredEmptyHFiles-1] regionserver.HRegion(1678): Closed testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:24:59,703 DEBUG [RegionOpenAndInitThread-testFIFOCompactionPolicyExpiredEmptyHFiles-1] regionserver.HRegion(1510): Region close journal: null at 1590978299703 Waiting for close lock at 1590978299703 Disabling compacts and flushes for region at 1590978299703 Disabling writes for close at 1590978299703 Writing region close event to WAL at 1590978299703 Closed at 1590978299703 2020-05-31 19:24:59,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] master.MasterRpcServices(1078): Checking to see if procedure is done procId=4 2020-05-31 19:24:59,813 INFO [ProcedureExecutor-3] hbase.MetaTableAccessor(1248): Added 1 2020-05-31 19:24:59,918 INFO [ProcedureExecutor-3] zookeeper.ZKTableStateManager(100): Moving table testFIFOCompactionPolicyExpiredEmptyHFiles state from null to ENABLING 2020-05-31 19:24:59,921 INFO [ProcedureExecutor-3] master.AssignmentManager(1758): Assigning 1 region(s) to foo.com,56775,1590978294064 2020-05-31 19:24:59,922 DEBUG [ProcedureExecutor-3] zookeeper.ZKAssign(174): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Async create of unassigned node 92bf95f6f53f0893e6ad400b3724872f with OFFLINE state 2020-05-31 19:24:59,922 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2020-05-31 19:24:59,922 DEBUG [main-EventThread] master.OfflineCallback(69): rs={92bf95f6f53f0893e6ad400b3724872f state=OFFLINE, ts=1590978299921, server=null}, server=foo.com,56775,1590978294064 2020-05-31 19:24:59,923 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={92bf95f6f53f0893e6ad400b3724872f state=OFFLINE, ts=1590978299921, server=null}, server=foo.com,56775,1590978294064 2020-05-31 19:24:59,927 DEBUG [ProcedureExecutor-3] master.AssignmentManager(1811): foo.com,56775,1590978294064 unassigned znodes=1 of total=1; oldCounter=0 2020-05-31 19:24:59,927 INFO [ProcedureExecutor-3] master.RegionStates(1309): Transition {92bf95f6f53f0893e6ad400b3724872f state=OFFLINE, ts=1590978299922, server=null} to {92bf95f6f53f0893e6ad400b3724872f state=PENDING_OPEN, ts=1590978299927, server=foo.com,56775,1590978294064} 2020-05-31 19:24:59,928 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=56775] regionserver.RSRpcServices(1876): Open testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:24:59,932 DEBUG [RS_OPEN_REGION-bvissaprag-ltm:56775-0] zookeeper.ZKAssign(805): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Transitioning 92bf95f6f53f0893e6ad400b3724872f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2020-05-31 19:24:59,933 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/92bf95f6f53f0893e6ad400b3724872f 2020-05-31 19:24:59,933 DEBUG [RS_OPEN_REGION-bvissaprag-ltm:56775-0] zookeeper.ZKAssign(888): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Transitioned node 92bf95f6f53f0893e6ad400b3724872f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2020-05-31 19:24:59,934 DEBUG [RS_OPEN_REGION-bvissaprag-ltm:56775-0] regionserver.HRegion(7076): Opening region: {ENCODED => 92bf95f6f53f0893e6ad400b3724872f, NAME => 'testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f.', STARTKEY => '', ENDKEY => ''} 2020-05-31 19:24:59,934 DEBUG [AM.ZK.Worker-pool5-t10] master.AssignmentManager(1062): Handling RS_ZK_REGION_OPENING, server=foo.com,56775,1590978294064, region=92bf95f6f53f0893e6ad400b3724872f, current_state={92bf95f6f53f0893e6ad400b3724872f state=PENDING_OPEN, ts=1590978299927, server=foo.com,56775,1590978294064} 2020-05-31 19:24:59,934 INFO [AM.ZK.Worker-pool5-t10] master.RegionStates(1309): Transition {92bf95f6f53f0893e6ad400b3724872f state=PENDING_OPEN, ts=1590978299927, server=foo.com,56775,1590978294064} to {92bf95f6f53f0893e6ad400b3724872f state=OPENING, ts=1590978299934, server=foo.com,56775,1590978294064} 2020-05-31 19:24:59,934 DEBUG [RS_OPEN_REGION-bvissaprag-ltm:56775-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testFIFOCompactionPolicyExpiredEmptyHFiles 92bf95f6f53f0893e6ad400b3724872f 2020-05-31 19:24:59,934 DEBUG [RS_OPEN_REGION-bvissaprag-ltm:56775-0] regionserver.HRegion(832): Instantiated testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:24:59,938 DEBUG [StoreOpener-92bf95f6f53f0893e6ad400b3724872f-1] util.CommonFSUtils(517): Set storagePolicy=HOT for path=hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f 2020-05-31 19:24:59,938 INFO [StoreOpener-92bf95f6f53f0893e6ad400b3724872f-1] hfile.CacheConfig(284): Created cacheConfig for f: blockCache=LruBlockCache{blockCount=0, currentSize=2238040, freeSize=3052054184, maxSize=3054292224, heapSize=2238040, minSize=2901577472, minFactor=0.95, multiSize=1450788736, multiFactor=0.5, singleSize=725394368, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2020-05-31 19:24:59,939 INFO [StoreOpener-92bf95f6f53f0893e6ad400b3724872f-1] compactions.CompactionConfiguration(149): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2020-05-31 19:24:59,939 INFO [StoreOpener-92bf95f6f53f0893e6ad400b3724872f-1] regionserver.HStore(377): Store=f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, encoding=NONE, compression=NONE 2020-05-31 19:24:59,941 DEBUG [RS_OPEN_REGION-bvissaprag-ltm:56775-0] regionserver.HRegion(4397): Found 0 recovered edits file(s) under hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f 2020-05-31 19:24:59,942 DEBUG [RS_OPEN_REGION-bvissaprag-ltm:56775-0] regionserver.HRegion(4397): Found 0 recovered edits file(s) under hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f 2020-05-31 19:24:59,945 DEBUG [RS_OPEN_REGION-bvissaprag-ltm:56775-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2020-05-31 19:24:59,947 DEBUG [RS_OPEN_REGION-bvissaprag-ltm:56775-0] wal.WALSplitter(750): Wrote region seqId=hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2020-05-31 19:24:59,947 INFO [RS_OPEN_REGION-bvissaprag-ltm:56775-0] regionserver.HRegion(988): Onlined 92bf95f6f53f0893e6ad400b3724872f; next sequenceid=2 2020-05-31 19:24:59,947 DEBUG [RS_OPEN_REGION-bvissaprag-ltm:56775-0] regionserver.HRegion(907): Region open journal: null at 1590978299934 Running coprocessor pre-open hook at 1590978299934 Writing region info on filesystem at 1590978299934 Initializing all the Stores at 1590978299936 (+2 ms) Instantiating store for column family {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => '1 SECOND', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1590978299936 Cleaning up temporary data from old regions at 1590978299942 (+6 ms) Cleaning up detritus from prior splits at 1590978299943 (+1 ms) Running coprocessor post-open hooks at 1590978299947 (+4 ms) Region opened successfully at 1590978299947 2020-05-31 19:24:59,948 DEBUG [RS_OPEN_REGION-bvissaprag-ltm:56775-0] zookeeper.ZKAssign(643): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Attempting to retransition opening state of node 92bf95f6f53f0893e6ad400b3724872f 2020-05-31 19:24:59,948 INFO [PostOpenDeployTasks:92bf95f6f53f0893e6ad400b3724872f] regionserver.HRegionServer(2108): Post open deploy tasks for testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:24:59,950 INFO [PostOpenDeployTasks:92bf95f6f53f0893e6ad400b3724872f] hbase.MetaTableAccessor(1448): Updated row testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. with server=foo.com,56775,1590978294064 2020-05-31 19:24:59,950 DEBUG [PostOpenDeployTasks:92bf95f6f53f0893e6ad400b3724872f] regionserver.HRegionServer(2142): Finished post open deploy task for testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:24:59,951 DEBUG [RS_OPEN_REGION-bvissaprag-ltm:56775-0] zookeeper.ZKAssign(805): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Transitioning 92bf95f6f53f0893e6ad400b3724872f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2020-05-31 19:24:59,951 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/92bf95f6f53f0893e6ad400b3724872f 2020-05-31 19:24:59,951 DEBUG [RS_OPEN_REGION-bvissaprag-ltm:56775-0] zookeeper.ZKAssign(888): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Transitioned node 92bf95f6f53f0893e6ad400b3724872f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2020-05-31 19:24:59,951 DEBUG [RS_OPEN_REGION-bvissaprag-ltm:56775-0] coordination.ZkOpenRegionCoordination(91): Transitioned 92bf95f6f53f0893e6ad400b3724872f to OPENED in zk on foo.com,56775,1590978294064 2020-05-31 19:24:59,951 DEBUG [RS_OPEN_REGION-bvissaprag-ltm:56775-0] handler.OpenRegionHandler(172): Opened testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. on foo.com,56775,1590978294064 2020-05-31 19:24:59,952 DEBUG [AM.ZK.Worker-pool5-t11] master.AssignmentManager(1062): Handling RS_ZK_REGION_OPENED, server=foo.com,56775,1590978294064, region=92bf95f6f53f0893e6ad400b3724872f, current_state={92bf95f6f53f0893e6ad400b3724872f state=OPENING, ts=1590978299934, server=foo.com,56775,1590978294064} 2020-05-31 19:24:59,952 INFO [AM.ZK.Worker-pool5-t11] master.RegionStates(1309): Transition {92bf95f6f53f0893e6ad400b3724872f state=OPENING, ts=1590978299934, server=foo.com,56775,1590978294064} to {92bf95f6f53f0893e6ad400b3724872f state=OPEN, ts=1590978299952, server=foo.com,56775,1590978294064} 2020-05-31 19:24:59,952 DEBUG [AM.ZK.Worker-pool5-t11] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 92bf95f6f53f0893e6ad400b3724872f from foo.com,56773,1590978293687; deleting unassigned node 2020-05-31 19:24:59,953 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/92bf95f6f53f0893e6ad400b3724872f 2020-05-31 19:24:59,953 DEBUG [AM.ZK.Worker-pool5-t11] zookeeper.ZKAssign(479): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Deleted unassigned node 92bf95f6f53f0893e6ad400b3724872f in expected state RS_ZK_REGION_OPENED 2020-05-31 19:24:59,953 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2020-05-31 19:24:59,953 DEBUG [AM.ZK.Worker-pool5-t12] master.AssignmentManager$5(1482): Znode testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. deleted, state: {92bf95f6f53f0893e6ad400b3724872f state=OPEN, ts=1590978299952, server=foo.com,56775,1590978294064} 2020-05-31 19:24:59,953 DEBUG [AM.ZK.Worker-pool5-t12] master.RegionStates(557): Onlined 92bf95f6f53f0893e6ad400b3724872f on foo.com,56775,1590978294064 2020-05-31 19:24:59,953 DEBUG [ProcedureExecutor-3] master.AssignmentManager(1953): Bulk assigning done for foo.com,56775,1590978294064 2020-05-31 19:24:59,953 INFO [ProcedureExecutor-3] zookeeper.ZKTableStateManager(100): Moving table testFIFOCompactionPolicyExpiredEmptyHFiles state from ENABLING to ENABLED 2020-05-31 19:25:00,277 DEBUG [ProcedureExecutor-3] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testFIFOCompactionPolicyExpiredEmptyHFiles/write-master:567730000000000 2020-05-31 19:25:00,277 DEBUG [ProcedureExecutor-3] procedure2.ProcedureExecutor(970): Procedure completed in 1.1130 sec: CreateTableProcedure (table=testFIFOCompactionPolicyExpiredEmptyHFiles) id=4 owner=foo state=FINISHED 2020-05-31 19:25:00,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] master.MasterRpcServices(1078): Checking to see if procedure is done procId=4 2020-05-31 19:25:00,296 DEBUG [htable-pool14-t1] ipc.RpcConnection(126): Use SIMPLE authentication for service ClientService, sasl=false 2020-05-31 19:25:00,297 DEBUG [htable-pool14-t1] ipc.BlockingRpcConnection(431): Connecting to foo.com/10.55.1.229:56775 2020-05-31 19:25:00,297 DEBUG [RpcServer.listener,port=56775] ipc.RpcServer$Listener(950): RpcServer.listener,port=56775: connection from 10.55.1.229:56801; # active connections: 3 2020-05-31 19:25:00,297 INFO [RpcServer.reader=1,bindAddress=foo.com,port=56775] ipc.RpcServer$Connection(1857): Auth successful for foo (auth:SIMPLE) 2020-05-31 19:25:00,298 INFO [RpcServer.reader=1,bindAddress=foo.com,port=56775] ipc.RpcServer$Connection(1887): Connection from 10.55.1.229 port: 56801 with version info: version: "1.7.0-SNAPSHOT" url: "git://foo.com/Users/foo/IdeaProjects/hbase-master" revision: "90b66544f0841f03bd0096d0679f526355a16fcd" user: "foo" date: "Sun May 31 14:23:50 PDT 2020" src_checksum: "9b20e6bd764436f9d419120b9f623653" version_major: 1 version_minor: 7 2020-05-31 19:25:00,304 INFO [main] client.HBaseAdmin$CreateTableFuture(798): Created testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,305 INFO [main] zookeeper.RecoverableZooKeeper(119): Process identifier=hconnection-0x6c518474 connecting to ZooKeeper ensemble=localhost:49339 2020-05-31 19:25:00,307 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): hconnection-0x6c5184740x0, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2020-05-31 19:25:00,308 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(683): hconnection-0x6c518474-0x1726db22d320008 connected 2020-05-31 19:25:00,309 DEBUG [main] ipc.AbstractRpcClient(199): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a066689, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2020-05-31 19:25:00,310 INFO [main] hbase.Waiter(183): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2020-05-31 19:25:00,311 DEBUG [htable-pool16-t1] ipc.RpcConnection(126): Use SIMPLE authentication for service ClientService, sasl=false 2020-05-31 19:25:00,312 DEBUG [htable-pool16-t1] ipc.BlockingRpcConnection(431): Connecting to foo.com/10.55.1.229:56775 2020-05-31 19:25:00,312 DEBUG [RpcServer.listener,port=56775] ipc.RpcServer$Listener(950): RpcServer.listener,port=56775: connection from 10.55.1.229:56803; # active connections: 4 2020-05-31 19:25:00,312 INFO [RpcServer.reader=2,bindAddress=foo.com,port=56775] ipc.RpcServer$Connection(1857): Auth successful for foo (auth:SIMPLE) 2020-05-31 19:25:00,313 INFO [RpcServer.reader=2,bindAddress=foo.com,port=56775] ipc.RpcServer$Connection(1887): Connection from 10.55.1.229 port: 56803 with version info: version: "1.7.0-SNAPSHOT" url: "git://foo.com/Users/foo/IdeaProjects/hbase-master" revision: "90b66544f0841f03bd0096d0679f526355a16fcd" user: "foo" date: "Sun May 31 14:23:50 PDT 2020" src_checksum: "9b20e6bd764436f9d419120b9f623653" version_major: 1 version_minor: 7 2020-05-31 19:25:00,315 INFO [main] client.ConnectionManager$HConnectionImplementation(1783): Closing zookeeper sessionid=0x1726db22d320008 2020-05-31 19:25:00,316 DEBUG [main] ipc.AbstractRpcClient(483): Stopping rpc client 2020-05-31 19:25:00,317 DEBUG [RpcServer.reader=2,bindAddress=foo.com,port=56775] ipc.RpcServer$Listener(982): RpcServer.listener,port=56775: DISCONNECTING client 10.55.1.229:56803 because read count=-1. Number of active connections: 4 2020-05-31 19:25:00,317 INFO [main] hbase.Waiter(183): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2020-05-31 19:25:00,318 INFO [main] zookeeper.RecoverableZooKeeper(119): Process identifier=hconnection-0x6d025d1d connecting to ZooKeeper ensemble=localhost:49339 2020-05-31 19:25:00,320 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): hconnection-0x6d025d1d0x0, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2020-05-31 19:25:00,320 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(683): hconnection-0x6d025d1d-0x1726db22d320009 connected 2020-05-31 19:25:00,321 DEBUG [main] ipc.AbstractRpcClient(199): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9e2ad91, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2020-05-31 19:25:00,322 DEBUG [hconnection-0x6d025d1d-metaLookup-shared--pool18-t1] ipc.RpcConnection(126): Use SIMPLE authentication for service ClientService, sasl=false 2020-05-31 19:25:00,322 DEBUG [hconnection-0x6d025d1d-metaLookup-shared--pool18-t1] ipc.BlockingRpcConnection(431): Connecting to foo.com/10.55.1.229:56775 2020-05-31 19:25:00,323 DEBUG [RpcServer.listener,port=56775] ipc.RpcServer$Listener(950): RpcServer.listener,port=56775: connection from 10.55.1.229:56805; # active connections: 4 2020-05-31 19:25:00,323 INFO [RpcServer.reader=0,bindAddress=foo.com,port=56775] ipc.RpcServer$Connection(1857): Auth successful for foo (auth:SIMPLE) 2020-05-31 19:25:00,323 INFO [RpcServer.reader=0,bindAddress=foo.com,port=56775] ipc.RpcServer$Connection(1887): Connection from 10.55.1.229 port: 56805 with version info: version: "1.7.0-SNAPSHOT" url: "git://foo.com/Users/foo/IdeaProjects/hbase-master" revision: "90b66544f0841f03bd0096d0679f526355a16fcd" user: "foo" date: "Sun May 31 14:23:50 PDT 2020" src_checksum: "9b20e6bd764436f9d419120b9f623653" version_major: 1 version_minor: 7 2020-05-31 19:25:00,340 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] master.MasterRpcServices(759): Client=foo//10.55.1.229 procedure request for: flush-table-proc 2020-05-31 19:25:00,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] procedure.ProcedureCoordinator(177): Submitting procedure testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,345 INFO [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' 2020-05-31 19:25:00,345 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2020-05-31 19:25:00,346 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'acquire' 2020-05-31 19:25:00,346 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles', kicking off acquire phase on members. 2020-05-31 19:25:00,346 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(420): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,346 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,347 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2020-05-31 19:25:00,347 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:00,347 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2020-05-31 19:25:00,347 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2020-05-31 19:25:00,347 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(420): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:00,347 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2020-05-31 19:25:00,347 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,347 DEBUG [main-EventThread] zookeeper.ZKUtil(420): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,347 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2020-05-31 19:25:00,347 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,347 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,349 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,349 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' with timeout 60000ms 2020-05-31 19:25:00,349 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2020-05-31 19:25:00,349 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'acquire' stage 2020-05-31 19:25:00,350 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2020-05-31 19:25:00,350 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool6-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:25:00,350 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2020-05-31 19:25:00,350 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool6-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. started... 2020-05-31 19:25:00,350 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool6-thread-1] regionserver.HRegion(2454): Flushing 1/1 column families, memstore=120 B 2020-05-31 19:25:00,361 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool6-thread-1] util.CommonFSUtils(517): Set storagePolicy=HOT for path=hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/.tmp/f 2020-05-31 19:25:00,361 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool6-thread-1] util.CommonFSUtils(517): Set storagePolicy=HOT for path=hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/.tmp/f 2020-05-31 19:25:00,798 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool6-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=6, memsize=120, hasBloomFilter=true, into tmp file hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/.tmp/f/25a116e7bd0c45d3ae7be216480a2942 2020-05-31 19:25:00,823 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool6-thread-1] regionserver.StoreFile(519): HFile Bloom filter type for 25a116e7bd0c45d3ae7be216480a2942: NONE, but ROW specified in column family configuration 2020-05-31 19:25:00,842 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool6-thread-1] regionserver.HRegionFileSystem(466): Committing store file hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/.tmp/f/25a116e7bd0c45d3ae7be216480a2942 as hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/25a116e7bd0c45d3ae7be216480a2942 2020-05-31 19:25:00,848 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool6-thread-1] regionserver.StoreFile(519): HFile Bloom filter type for 25a116e7bd0c45d3ae7be216480a2942: NONE, but ROW specified in column family configuration 2020-05-31 19:25:00,848 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool6-thread-1] regionserver.HStore(1075): Added hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/25a116e7bd0c45d3ae7be216480a2942, entries=0, sequenceid=6, filesize=4.5 K 2020-05-31 19:25:00,850 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool6-thread-1] regionserver.HRegion(2753): Finished memstore flush of ~120 B/120, currentsize=0 B/0 for region testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. in 500ms, sequenceid=6, compaction requested=true 2020-05-31 19:25:00,850 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool6-thread-1] regionserver.HRegion(2254): Flush status journal: Acquiring readlock on region at 1590978300350 Running coprocessor pre-flush hooks at 1590978300350 Obtaining lock to block concurrent updates at 1590978300350 Preparing to flush by snapshotting stores in 92bf95f6f53f0893e6ad400b3724872f at 1590978300350 Finished memstore snapshotting testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f., syncing WAL and waiting on mvcc, flushsize=120 at 1590978300356 (+6 ms) Flushing stores of testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. at 1590978300357 (+1 ms) Flushing f: creating writer at 1590978300358 (+1 ms) Flushing f: appending metadata at 1590978300378 (+20 ms) Flushing f: closing flushed file at 1590978300380 (+2 ms) Flushing f: reopening flushed file at 1590978300842 (+462 ms) Finished memstore flush of ~120 B/120, currentsize=0 B/0 for region testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. in 500ms, sequenceid=6, compaction requested=true at 1590978300850 (+8 ms) Running post-flush coprocessor hooks at 1590978300850 Flush successful at 1590978300850 2020-05-31 19:25:00,850 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool6-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:25:00,851 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2020-05-31 19:25:00,851 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2020-05-31 19:25:00,851 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks foo.com,56775,1590978294064 2020-05-31 19:25:00,851 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' locally acquired 2020-05-31 19:25:00,851 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'foo.com,56775,1590978294064' joining acquired barrier for procedure (testFIFOCompactionPolicyExpiredEmptyHFiles) in zk 2020-05-31 19:25:00,852 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:00,852 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,852 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:00,852 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2020-05-31 19:25:00,852 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2020-05-31 19:25:00,852 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] zookeeper.ZKUtil(420): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,852 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2020-05-31 19:25:00,852 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2020-05-31 19:25:00,853 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2020-05-31 19:25:00,853 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,853 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------foo.com,56775,1590978294064 2020-05-31 19:25:00,853 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2020-05-31 19:25:00,853 DEBUG [main-EventThread] procedure.Procedure(298): member: 'foo.com,56775,1590978294064' joining acquired barrier for procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' on coordinator 2020-05-31 19:25:00,853 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'in-barrier' execution. 2020-05-31 19:25:00,853 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@36a03572[Count = 0] remaining members to acquire global barrier 2020-05-31 19:25:00,853 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,854 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,854 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,854 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,854 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(420): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:00,854 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' received 'reached' from coordinator. 2020-05-31 19:25:00,854 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2020-05-31 19:25:00,854 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' locally completed 2020-05-31 19:25:00,854 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' completed for member 'foo.com,56775,1590978294064' in zk 2020-05-31 19:25:00,855 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:00,855 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' has notified controller of completion 2020-05-31 19:25:00,855 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2020-05-31 19:25:00,855 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:00,855 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2020-05-31 19:25:00,855 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' completed. 2020-05-31 19:25:00,856 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2020-05-31 19:25:00,856 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2020-05-31 19:25:00,856 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2020-05-31 19:25:00,856 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,857 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------foo.com,56775,1590978294064 2020-05-31 19:25:00,857 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2020-05-31 19:25:00,857 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,857 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------foo.com,56775,1590978294064 2020-05-31 19:25:00,857 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' member 'foo.com,56775,1590978294064': 2020-05-31 19:25:00,857 INFO [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' execution completed 2020-05-31 19:25:00,857 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'foo.com,56775,1590978294064' released barrier for procedure'testFIFOCompactionPolicyExpiredEmptyHFiles', counting down latch. Waiting for 0 more 2020-05-31 19:25:00,857 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2020-05-31 19:25:00,858 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2020-05-31 19:25:00,858 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,858 INFO [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure testFIFOCompactionPolicyExpiredEmptyHFilesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2020-05-31 19:25:00,858 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,858 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,859 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,859 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,859 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2020-05-31 19:25:00,859 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2020-05-31 19:25:00,859 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,859 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(418): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:00,859 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2020-05-31 19:25:00,859 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2020-05-31 19:25:00,859 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2020-05-31 19:25:00,859 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2020-05-31 19:25:00,859 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,859 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,859 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2020-05-31 19:25:00,859 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,859 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(418): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:00,860 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------foo.com,56775,1590978294064 2020-05-31 19:25:00,860 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2020-05-31 19:25:00,860 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,860 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------foo.com,56775,1590978294064 2020-05-31 19:25:00,864 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:00,864 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,864 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2020-05-31 19:25:00,864 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,864 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2020-05-31 19:25:00,864 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2020-05-31 19:25:00,864 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:00,864 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,864 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,864 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,864 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2020-05-31 19:25:00,864 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'testFIFOCompactionPolicyExpiredEmptyHFiles' 2020-05-31 19:25:00,864 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2020-05-31 19:25:00,865 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2020-05-31 19:25:00,865 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2020-05-31 19:25:00,865 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2020-05-31 19:25:00,866 DEBUG [main] client.HBaseAdmin(4173): Waiting a max of 300000 ms for procedure 'flush-table-proc : testFIFOCompactionPolicyExpiredEmptyHFiles'' to complete. (max 8571 ms per retry) 2020-05-31 19:25:00,866 DEBUG [main] client.HBaseAdmin(4182): (#1) Sleeping: 100ms while waiting for procedure completion. 2020-05-31 19:25:00,970 DEBUG [main] client.HBaseAdmin(4188): Getting current status of procedure from master... 2020-05-31 19:25:00,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] master.MasterRpcServices(1008): Checking to see if procedure from request:flush-table-proc is done 2020-05-31 19:25:00,982 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] master.MasterRpcServices(759): Client=foo//10.55.1.229 procedure request for: flush-table-proc 2020-05-31 19:25:00,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] procedure.ProcedureCoordinator(155): Procedure testFIFOCompactionPolicyExpiredEmptyHFiles was in running list but was completed. Accepting new attempt. 2020-05-31 19:25:00,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] procedure.ProcedureCoordinator(177): Submitting procedure testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,984 INFO [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' 2020-05-31 19:25:00,984 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2020-05-31 19:25:00,984 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'acquire' 2020-05-31 19:25:00,984 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles', kicking off acquire phase on members. 2020-05-31 19:25:00,985 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(420): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,985 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,986 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2020-05-31 19:25:00,986 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:00,986 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2020-05-31 19:25:00,986 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2020-05-31 19:25:00,986 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(420): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:00,986 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2020-05-31 19:25:00,986 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,986 DEBUG [main-EventThread] zookeeper.ZKUtil(420): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,986 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2020-05-31 19:25:00,986 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,986 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,987 WARN [main-EventThread] procedure.ProcedureMember(142): A completed old subproc testFIFOCompactionPolicyExpiredEmptyHFiles is still present, removing 2020-05-31 19:25:00,987 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:00,987 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' with timeout 60000ms 2020-05-31 19:25:00,987 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2020-05-31 19:25:00,987 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'acquire' stage 2020-05-31 19:25:00,987 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2020-05-31 19:25:00,987 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2020-05-31 19:25:00,987 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool7-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:25:00,987 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool7-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. started... 2020-05-31 19:25:00,988 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool7-thread-1] regionserver.HRegion(2454): Flushing 1/1 column families, memstore=120 B 2020-05-31 19:25:00,990 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool7-thread-1] util.CommonFSUtils(517): Set storagePolicy=HOT for path=hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/.tmp/f 2020-05-31 19:25:00,990 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool7-thread-1] util.CommonFSUtils(517): Set storagePolicy=HOT for path=hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/.tmp/f 2020-05-31 19:25:01,402 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool7-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=11, memsize=120, hasBloomFilter=true, into tmp file hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/.tmp/f/6911ffbbcb964d57a76f847ca0cdb616 2020-05-31 19:25:01,410 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool7-thread-1] regionserver.StoreFile(519): HFile Bloom filter type for 6911ffbbcb964d57a76f847ca0cdb616: NONE, but ROW specified in column family configuration 2020-05-31 19:25:01,412 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool7-thread-1] regionserver.HRegionFileSystem(466): Committing store file hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/.tmp/f/6911ffbbcb964d57a76f847ca0cdb616 as hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616 2020-05-31 19:25:01,419 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool7-thread-1] regionserver.StoreFile(519): HFile Bloom filter type for 6911ffbbcb964d57a76f847ca0cdb616: NONE, but ROW specified in column family configuration 2020-05-31 19:25:01,419 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool7-thread-1] regionserver.HStore(1075): Added hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616, entries=0, sequenceid=11, filesize=4.5 K 2020-05-31 19:25:01,420 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool7-thread-1] regionserver.HRegion(2753): Finished memstore flush of ~120 B/120, currentsize=0 B/0 for region testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. in 432ms, sequenceid=11, compaction requested=true 2020-05-31 19:25:01,420 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool7-thread-1] regionserver.HRegion(2254): Flush status journal: Acquiring readlock on region at 1590978300988 Running coprocessor pre-flush hooks at 1590978300988 Obtaining lock to block concurrent updates at 1590978300988 Preparing to flush by snapshotting stores in 92bf95f6f53f0893e6ad400b3724872f at 1590978300988 Finished memstore snapshotting testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f., syncing WAL and waiting on mvcc, flushsize=120 at 1590978300988 Flushing stores of testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. at 1590978300988 Flushing f: creating writer at 1590978300989 (+1 ms) Flushing f: appending metadata at 1590978300992 (+3 ms) Flushing f: closing flushed file at 1590978300992 Flushing f: reopening flushed file at 1590978301412 (+420 ms) Finished memstore flush of ~120 B/120, currentsize=0 B/0 for region testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. in 432ms, sequenceid=11, compaction requested=true at 1590978301420 (+8 ms) Running post-flush coprocessor hooks at 1590978301420 Flush successful at 1590978301420 2020-05-31 19:25:01,420 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool7-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:25:01,420 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2020-05-31 19:25:01,420 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2020-05-31 19:25:01,420 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks foo.com,56775,1590978294064 2020-05-31 19:25:01,421 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' locally acquired 2020-05-31 19:25:01,421 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'foo.com,56775,1590978294064' joining acquired barrier for procedure (testFIFOCompactionPolicyExpiredEmptyHFiles) in zk 2020-05-31 19:25:01,422 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,422 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,422 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,422 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2020-05-31 19:25:01,422 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2020-05-31 19:25:01,422 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] zookeeper.ZKUtil(420): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,422 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2020-05-31 19:25:01,422 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2020-05-31 19:25:01,422 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2020-05-31 19:25:01,423 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,423 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------foo.com,56775,1590978294064 2020-05-31 19:25:01,423 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2020-05-31 19:25:01,423 DEBUG [main-EventThread] procedure.Procedure(298): member: 'foo.com,56775,1590978294064' joining acquired barrier for procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' on coordinator 2020-05-31 19:25:01,423 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@8f30efa[Count = 0] remaining members to acquire global barrier 2020-05-31 19:25:01,423 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'in-barrier' execution. 2020-05-31 19:25:01,423 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,424 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,424 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,424 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,424 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' received 'reached' from coordinator. 2020-05-31 19:25:01,424 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' locally completed 2020-05-31 19:25:01,424 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' completed for member 'foo.com,56775,1590978294064' in zk 2020-05-31 19:25:01,424 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(420): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,424 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2020-05-31 19:25:01,425 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,425 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' has notified controller of completion 2020-05-31 19:25:01,425 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2020-05-31 19:25:01,425 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,425 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' completed. 2020-05-31 19:25:01,425 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2020-05-31 19:25:01,425 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2020-05-31 19:25:01,426 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2020-05-31 19:25:01,426 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2020-05-31 19:25:01,426 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,426 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------foo.com,56775,1590978294064 2020-05-31 19:25:01,426 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2020-05-31 19:25:01,427 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,427 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------foo.com,56775,1590978294064 2020-05-31 19:25:01,427 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' member 'foo.com,56775,1590978294064': 2020-05-31 19:25:01,427 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'foo.com,56775,1590978294064' released barrier for procedure'testFIFOCompactionPolicyExpiredEmptyHFiles', counting down latch. Waiting for 0 more 2020-05-31 19:25:01,427 INFO [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' execution completed 2020-05-31 19:25:01,427 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2020-05-31 19:25:01,427 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2020-05-31 19:25:01,427 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,427 INFO [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure testFIFOCompactionPolicyExpiredEmptyHFilesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2020-05-31 19:25:01,428 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,428 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,428 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,428 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,428 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2020-05-31 19:25:01,428 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2020-05-31 19:25:01,428 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,428 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(418): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,428 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2020-05-31 19:25:01,428 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2020-05-31 19:25:01,428 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2020-05-31 19:25:01,428 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2020-05-31 19:25:01,429 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,429 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,429 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2020-05-31 19:25:01,429 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,429 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(418): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,429 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------foo.com,56775,1590978294064 2020-05-31 19:25:01,429 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2020-05-31 19:25:01,430 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,430 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------foo.com,56775,1590978294064 2020-05-31 19:25:01,431 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2020-05-31 19:25:01,432 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2020-05-31 19:25:01,432 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2020-05-31 19:25:01,432 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2020-05-31 19:25:01,432 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'testFIFOCompactionPolicyExpiredEmptyHFiles' 2020-05-31 19:25:01,432 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2020-05-31 19:25:01,432 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2020-05-31 19:25:01,432 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2020-05-31 19:25:01,432 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2020-05-31 19:25:01,432 DEBUG [main-EventThread] zookeeper.ZKUtil(677): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles because node does not exist (not an error) 2020-05-31 19:25:01,432 DEBUG [main] client.HBaseAdmin(4173): Waiting a max of 300000 ms for procedure 'flush-table-proc : testFIFOCompactionPolicyExpiredEmptyHFiles'' to complete. (max 8571 ms per retry) 2020-05-31 19:25:01,432 DEBUG [main] client.HBaseAdmin(4182): (#1) Sleeping: 100ms while waiting for procedure completion. 2020-05-31 19:25:01,432 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,433 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,433 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,433 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,433 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,433 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,433 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,535 DEBUG [main] client.HBaseAdmin(4188): Getting current status of procedure from master... 2020-05-31 19:25:01,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] master.MasterRpcServices(1008): Checking to see if procedure from request:flush-table-proc is done 2020-05-31 19:25:01,541 INFO [main] zookeeper.RecoverableZooKeeper(119): Process identifier=hbase-admin-on-hconnection-0x66c38e51 connecting to ZooKeeper ensemble=localhost:49339 2020-05-31 19:25:01,544 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): hbase-admin-on-hconnection-0x66c38e510x0, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2020-05-31 19:25:01,545 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(683): hbase-admin-on-hconnection-0x66c38e51-0x1726db22d32000a connected 2020-05-31 19:25:01,546 DEBUG [main] ipc.RpcConnection(126): Use SIMPLE authentication for service AdminService, sasl=false 2020-05-31 19:25:01,546 DEBUG [main] ipc.BlockingRpcConnection(431): Connecting to foo.com/10.55.1.229:56775 2020-05-31 19:25:01,547 DEBUG [RpcServer.listener,port=56775] ipc.RpcServer$Listener(950): RpcServer.listener,port=56775: connection from 10.55.1.229:56809; # active connections: 5 2020-05-31 19:25:01,547 INFO [RpcServer.reader=1,bindAddress=foo.com,port=56775] ipc.RpcServer$Connection(1857): Auth successful for foo (auth:SIMPLE) 2020-05-31 19:25:01,547 INFO [RpcServer.reader=1,bindAddress=foo.com,port=56775] ipc.RpcServer$Connection(1887): Connection from 10.55.1.229 port: 56809 with version info: version: "1.7.0-SNAPSHOT" url: "git://foo.com/Users/foo/IdeaProjects/hbase-master" revision: "90b66544f0841f03bd0096d0679f526355a16fcd" user: "foo" date: "Sun May 31 14:23:50 PDT 2020" src_checksum: "9b20e6bd764436f9d419120b9f623653" version_major: 1 version_minor: 7 2020-05-31 19:25:01,548 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=56775] regionserver.RSRpcServices(1516): Compacting testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:25:01,548 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=56775] regionserver.DefaultStoreFileManager(204): Found an expired store file: hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/25a116e7bd0c45d3ae7be216480a2942 whose maxTimeStamp is 1590978290321, which is below 1590978300548 2020-05-31 19:25:01,552 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=56775] regionserver.HStore(1971): Completing compaction... 2020-05-31 19:25:01,552 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=56775] regionserver.HStore(1890): Completed removal of 1 unnecessary (expired) file(s) in f of testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f.; total size for store is 4.5 K 2020-05-31 19:25:01,552 WARN [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=56775] compactions.FIFOCompactionPolicy(63): Major compaction is not supported for FIFO compaction policy. Ignore the flag. 2020-05-31 19:25:01,553 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=56775] regionserver.HStore(1845): 92bf95f6f53f0893e6ad400b3724872f - f: Initiating minor compaction 2020-05-31 19:25:01,554 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=56775] regionserver.CompactSplitThread(405): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@6c676919; Because: User-triggered major compaction; compaction_queue=(0:0), split_queue=0, merge_queue=0 2020-05-31 19:25:01,554 INFO [RS:0;bvissaprag-ltm:56775-shortCompactions-0] regionserver.HRegion(2115): Starting compaction on f in region testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:25:01,555 INFO [RS:0;bvissaprag-ltm:56775-shortCompactions-0] regionserver.HStore(1422): Starting compaction of 1 file(s) in f of testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. into tmpdir=hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/.tmp, totalSize=4.5 K 2020-05-31 19:25:01,555 DEBUG [RS:0;bvissaprag-ltm:56775-shortCompactions-0] compactions.Compactor(202): Compacting hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616, keycount=0, bloomtype=NONE, size=4.5 K, encoding=NONE, compression=NONE, seqNum=11 2020-05-31 19:25:01,563 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] master.MasterRpcServices(759): Client=foo//10.55.1.229 procedure request for: flush-table-proc 2020-05-31 19:25:01,564 INFO [RS:0;bvissaprag-ltm:56775-shortCompactions-0] regionserver.StoreFile(519): HFile Bloom filter type for 6911ffbbcb964d57a76f847ca0cdb616: NONE, but ROW specified in column family configuration 2020-05-31 19:25:01,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] procedure.ProcedureCoordinator(155): Procedure testFIFOCompactionPolicyExpiredEmptyHFiles was in running list but was completed. Accepting new attempt. 2020-05-31 19:25:01,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] procedure.ProcedureCoordinator(177): Submitting procedure testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,565 INFO [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' 2020-05-31 19:25:01,565 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2020-05-31 19:25:01,566 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'acquire' 2020-05-31 19:25:01,566 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles', kicking off acquire phase on members. 2020-05-31 19:25:01,566 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(420): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,566 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,567 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,567 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2020-05-31 19:25:01,567 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2020-05-31 19:25:01,567 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2020-05-31 19:25:01,567 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(420): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,567 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2020-05-31 19:25:01,567 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,567 DEBUG [main-EventThread] zookeeper.ZKUtil(420): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,568 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2020-05-31 19:25:01,568 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,568 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,568 WARN [main-EventThread] procedure.ProcedureMember(142): A completed old subproc testFIFOCompactionPolicyExpiredEmptyHFiles is still present, removing 2020-05-31 19:25:01,568 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,568 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' with timeout 60000ms 2020-05-31 19:25:01,568 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2020-05-31 19:25:01,569 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'acquire' stage 2020-05-31 19:25:01,569 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2020-05-31 19:25:01,569 DEBUG [RS:0;bvissaprag-ltm:56775-shortCompactions-0] util.CommonFSUtils(517): Set storagePolicy=HOT for path=hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/.tmp/f 2020-05-31 19:25:01,569 DEBUG [RS:0;bvissaprag-ltm:56775-shortCompactions-0] util.CommonFSUtils(517): Set storagePolicy=HOT for path=hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/.tmp/f 2020-05-31 19:25:01,569 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool8-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:25:01,569 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2020-05-31 19:25:01,569 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool8-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. started... 2020-05-31 19:25:01,570 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool8-thread-1] regionserver.HRegion(2454): Flushing 1/1 column families, memstore=120 B 2020-05-31 19:25:01,570 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2020-05-31 19:25:01,572 INFO [RS:0;bvissaprag-ltm:56775-shortCompactions-0] throttle.PressureAwareThroughputController(145): testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f.#f#compaction#2 average throughput is NaN MB/sec, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/sec 2020-05-31 19:25:01,572 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool8-thread-1] util.CommonFSUtils(517): Set storagePolicy=HOT for path=hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/.tmp/f 2020-05-31 19:25:01,572 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool8-thread-1] util.CommonFSUtils(517): Set storagePolicy=HOT for path=hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/.tmp/f 2020-05-31 19:25:01,588 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool8-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=17, memsize=120, hasBloomFilter=true, into tmp file hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/.tmp/f/6ad60489e49b44e59af08d4c13f5ec24 2020-05-31 19:25:01,599 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool8-thread-1] regionserver.StoreFile(519): HFile Bloom filter type for 6ad60489e49b44e59af08d4c13f5ec24: NONE, but ROW specified in column family configuration 2020-05-31 19:25:01,602 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool8-thread-1] regionserver.HRegionFileSystem(466): Committing store file hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/.tmp/f/6ad60489e49b44e59af08d4c13f5ec24 as hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6ad60489e49b44e59af08d4c13f5ec24 2020-05-31 19:25:01,613 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool8-thread-1] regionserver.StoreFile(519): HFile Bloom filter type for 6ad60489e49b44e59af08d4c13f5ec24: NONE, but ROW specified in column family configuration 2020-05-31 19:25:01,613 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool8-thread-1] regionserver.HStore(1075): Added hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6ad60489e49b44e59af08d4c13f5ec24, entries=0, sequenceid=17, filesize=4.5 K 2020-05-31 19:25:01,614 INFO [rs(foo.com,56775,1590978294064)-flush-proc-pool8-thread-1] regionserver.HRegion(2753): Finished memstore flush of ~120 B/120, currentsize=0 B/0 for region testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. in 44ms, sequenceid=17, compaction requested=true 2020-05-31 19:25:01,615 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool8-thread-1] regionserver.HRegion(2254): Flush status journal: Acquiring readlock on region at 1590978301570 Running coprocessor pre-flush hooks at 1590978301570 Obtaining lock to block concurrent updates at 1590978301570 Preparing to flush by snapshotting stores in 92bf95f6f53f0893e6ad400b3724872f at 1590978301570 Finished memstore snapshotting testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f., syncing WAL and waiting on mvcc, flushsize=120 at 1590978301570 Flushing stores of testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. at 1590978301570 Flushing f: creating writer at 1590978301570 Flushing f: appending metadata at 1590978301575 (+5 ms) Flushing f: closing flushed file at 1590978301575 Flushing f: reopening flushed file at 1590978301604 (+29 ms) Finished memstore flush of ~120 B/120, currentsize=0 B/0 for region testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. in 44ms, sequenceid=17, compaction requested=true at 1590978301614 (+10 ms) Running post-flush coprocessor hooks at 1590978301615 (+1 ms) Flush successful at 1590978301615 2020-05-31 19:25:01,615 DEBUG [rs(foo.com,56775,1590978294064)-flush-proc-pool8-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:25:01,615 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2020-05-31 19:25:01,615 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2020-05-31 19:25:01,615 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks foo.com,56775,1590978294064 2020-05-31 19:25:01,615 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' locally acquired 2020-05-31 19:25:01,615 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'foo.com,56775,1590978294064' joining acquired barrier for procedure (testFIFOCompactionPolicyExpiredEmptyHFiles) in zk 2020-05-31 19:25:01,616 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,616 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,616 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,617 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2020-05-31 19:25:01,617 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2020-05-31 19:25:01,617 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] zookeeper.ZKUtil(420): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,617 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2020-05-31 19:25:01,617 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2020-05-31 19:25:01,617 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2020-05-31 19:25:01,619 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,620 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------foo.com,56775,1590978294064 2020-05-31 19:25:01,621 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2020-05-31 19:25:01,622 DEBUG [main-EventThread] procedure.Procedure(298): member: 'foo.com,56775,1590978294064' joining acquired barrier for procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' on coordinator 2020-05-31 19:25:01,622 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@15303ecd[Count = 0] remaining members to acquire global barrier 2020-05-31 19:25:01,622 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'in-barrier' execution. 2020-05-31 19:25:01,622 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,622 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,622 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,622 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,623 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' received 'reached' from coordinator. 2020-05-31 19:25:01,623 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' locally completed 2020-05-31 19:25:01,623 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' completed for member 'foo.com,56775,1590978294064' in zk 2020-05-31 19:25:01,623 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(420): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,623 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2020-05-31 19:25:01,623 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,623 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' has notified controller of completion 2020-05-31 19:25:01,623 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2020-05-31 19:25:01,623 DEBUG [member: 'foo.com,56775,1590978294064' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' completed. 2020-05-31 19:25:01,623 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,624 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2020-05-31 19:25:01,624 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2020-05-31 19:25:01,624 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2020-05-31 19:25:01,625 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2020-05-31 19:25:01,625 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,625 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------foo.com,56775,1590978294064 2020-05-31 19:25:01,625 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2020-05-31 19:25:01,626 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,626 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------foo.com,56775,1590978294064 2020-05-31 19:25:01,626 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' member 'foo.com,56775,1590978294064': 2020-05-31 19:25:01,626 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'foo.com,56775,1590978294064' released barrier for procedure'testFIFOCompactionPolicyExpiredEmptyHFiles', counting down latch. Waiting for 0 more 2020-05-31 19:25:01,626 INFO [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' execution completed 2020-05-31 19:25:01,626 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2020-05-31 19:25:01,627 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2020-05-31 19:25:01,627 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,627 INFO [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure testFIFOCompactionPolicyExpiredEmptyHFilesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2020-05-31 19:25:01,628 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,628 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,628 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,628 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,628 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,628 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2020-05-31 19:25:01,628 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2020-05-31 19:25:01,628 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2020-05-31 19:25:01,628 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2020-05-31 19:25:01,628 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2020-05-31 19:25:01,628 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(418): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,628 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2020-05-31 19:25:01,628 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,629 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,629 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2020-05-31 19:25:01,629 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,629 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(418): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,629 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------foo.com,56775,1590978294064 2020-05-31 19:25:01,630 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2020-05-31 19:25:01,630 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,630 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------foo.com,56775,1590978294064 2020-05-31 19:25:01,631 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2020-05-31 19:25:01,631 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2020-05-31 19:25:01,631 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2020-05-31 19:25:01,631 DEBUG [(foo.com,56773,1590978293687)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2020-05-31 19:25:01,631 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'testFIFOCompactionPolicyExpiredEmptyHFiles' 2020-05-31 19:25:01,632 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2020-05-31 19:25:01,631 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2020-05-31 19:25:01,632 DEBUG [main-EventThread] zookeeper.ZKUtil(677): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles because node does not exist (not an error) 2020-05-31 19:25:01,632 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,632 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): regionserver:56775-0x1726db22d320001, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2020-05-31 19:25:01,632 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,632 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,632 DEBUG [main] client.HBaseAdmin(4173): Waiting a max of 300000 ms for procedure 'flush-table-proc : testFIFOCompactionPolicyExpiredEmptyHFiles'' to complete. (max 8571 ms per retry) 2020-05-31 19:25:01,632 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2020-05-31 19:25:01,632 DEBUG [main] client.HBaseAdmin(4182): (#1) Sleeping: 100ms while waiting for procedure completion. 2020-05-31 19:25:01,632 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/foo.com,56775,1590978294064 2020-05-31 19:25:01,632 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,632 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,632 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2020-05-31 19:25:01,632 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2020-05-31 19:25:01,632 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): master:56773-0x1726db22d320000, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2020-05-31 19:25:01,632 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2020-05-31 19:25:01,735 DEBUG [main] client.HBaseAdmin(4188): Getting current status of procedure from master... 2020-05-31 19:25:01,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56773] master.MasterRpcServices(1008): Checking to see if procedure from request:flush-table-proc is done 2020-05-31 19:25:01,738 INFO [main] zookeeper.RecoverableZooKeeper(119): Process identifier=hbase-admin-on-hconnection-0x66c38e51 connecting to ZooKeeper ensemble=localhost:49339 2020-05-31 19:25:01,741 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=56775] regionserver.RSRpcServices(1516): Compacting testFIFOCompactionPolicyExpiredEmptyHFiles,,1590978299053.92bf95f6f53f0893e6ad400b3724872f. 2020-05-31 19:25:01,741 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(620): hbase-admin-on-hconnection-0x66c38e510x0, quorum=localhost:49339, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2020-05-31 19:25:01,741 ERROR [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=56775] ipc.RpcServer(2462): Unexpected throwable object java.lang.IllegalArgumentException: [hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616, hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6ad60489e49b44e59af08d4c13f5ec24] overlaps with [hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616] at com.google.common.base.Preconditions.checkArgument(Preconditions.java:119) at org.apache.hadoop.hbase.regionserver.HStore.addToCompactingFiles(HStore.java:1857) at org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1815) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:420) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:393) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:322) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:311) at org.apache.hadoop.hbase.regionserver.RSRpcServices.compactRegion(RSRpcServices.java:1549) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:23649) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2405) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:311) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:291) 2020-05-31 19:25:01,741 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(683): hbase-admin-on-hconnection-0x66c38e51-0x1726db22d32000b connected 2020-05-31 19:25:01,742 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=56775] ipc.CallRunner(130): RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=56775: callId: 37 service: AdminService methodName: CompactRegion size: 126 connection: 10.55.1.229:56809 deadline: 1590978361741 java.io.IOException: [hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616, hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6ad60489e49b44e59af08d4c13f5ec24] overlaps with [hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616] at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2463) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:311) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:291) Caused by: java.lang.IllegalArgumentException: [hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616, hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6ad60489e49b44e59af08d4c13f5ec24] overlaps with [hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616] at com.google.common.base.Preconditions.checkArgument(Preconditions.java:119) at org.apache.hadoop.hbase.regionserver.HStore.addToCompactingFiles(HStore.java:1857) at org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1815) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:420) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:393) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:322) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:311) at org.apache.hadoop.hbase.regionserver.RSRpcServices.compactRegion(RSRpcServices.java:1549) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:23649) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2405) ... 3 more 2020-05-31 19:25:01,767 INFO [main] hbase.HBaseTestingUtility(1172): Shutting down minicluster 2020-05-31 19:25:01,768 INFO [main] client.ConnectionManager$HConnectionImplementation(2280): Closing master protocol: MasterService java.io.IOException: java.io.IOException: [hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616, hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6ad60489e49b44e59af08d4c13f5ec24] overlaps with [hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616] at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2463) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:311) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:291) Caused by: java.lang.IllegalArgumentException: [hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616, hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6ad60489e49b44e59af08d4c13f5ec24] overlaps with [hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616] at com.google.common.base.Preconditions.checkArgument(Preconditions.java:119) at org.apache.hadoop.hbase.regionserver.HStore.addToCompactingFiles(HStore.java:1857) at org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1815) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:420) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:393) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:322) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:311) at org.apache.hadoop.hbase.regionserver.RSRpcServices.compactRegion(RSRpcServices.java:1549) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:23649) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2405) ... 3 more at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:95) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:85) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:372) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:334) at org.apache.hadoop.hbase.client.HBaseAdmin.compact(HBaseAdmin.java:2270) at org.apache.hadoop.hbase.client.HBaseAdmin.compact(HBaseAdmin.java:2221) at org.apache.hadoop.hbase.client.HBaseAdmin.majorCompact(HBaseAdmin.java:2118) at org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy.testFIFOCompactionPolicyExpiredEmptyHFiles(TestFIFOCompactionPolicy.java:236) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:67) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: [hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616, hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6ad60489e49b44e59af08d4c13f5ec24] overlaps with [hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616] at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2463) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:311) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:291) Caused by: java.lang.IllegalArgumentException: [hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616, hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6ad60489e49b44e59af08d4c13f5ec24] overlaps with [hdfs://localhost:56764/user/foo/test-data/b19a7255-82f2-4935-9d7a-7f640a45c202/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/92bf95f6f53f0893e6ad400b3724872f/f/6911ffbbcb964d57a76f847ca0cdb616] at com.google.common.base.Preconditions.checkArgument(Preconditions.java:119) at org.apache.hadoop.hbase.regionserver.HStore.addToCompactingFiles(HStore.java:1857) at org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1815) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:420) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:393) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:322) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:311) at org.apache.hadoop.hbase.regionserver.RSRpcServices.compactRegion(RSRpcServices.java:1549) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:23649) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2405) ... 3 more at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.readResponse(BlockingRpcConnection.java:600) at org.apache.hadoop.hbase.ipc.BlockingRpcConnection.run(BlockingRpcConnection.java:334) at java.lang.Thread.run(Thread.java:748)