2017-10-03 21:45:36,021 INFO [main] hbase.ResourceChecker(148): before: coprocessor.TestRegionObserverInterface#testRecovery Thread=279, OpenFileDescriptor=1145, MaxFileDescriptor=32000, SystemLoadAverage=710, ProcessCount=347, AvailableMemoryMB=47494 2017-10-03 21:45:36,021 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1145 is superior to 1024 2017-10-03 21:45:36,022 INFO [Time-limited test] coprocessor.TestRegionObserverInterface(582): org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testRecovery 2017-10-03 21:45:36,029 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$3(1748): Client=hbase//172.18.128.12 create 'TestTable.testRecovery', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2017-10-03 21:45:36,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=27, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestTable.testRecovery 2017-10-03 21:45:36,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=27 2017-10-03 21:45:36,383 INFO [IPC Server handler 0 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741846_1022{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|FINALIZED]]} size 0 2017-10-03 21:45:36,388 DEBUG [ProcExecWrkr-13] util.FSTableDescriptors(728): Wrote descriptor into: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRecovery/.tabledesc/.tableinfo.0000000001 2017-10-03 21:45:36,390 INFO [RegionOpenAndInitThread-TestTable.testRecovery-1] regionserver.HRegion(6477): creating HRegion TestTable.testRecovery HTD == 'TestTable.testRecovery', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp Table name == TestTable.testRecovery 2017-10-03 21:45:36,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=27 2017-10-03 21:45:36,401 INFO [IPC Server handler 1 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741847_1023{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|RBW]]} size 57 2017-10-03 21:45:36,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=27 2017-10-03 21:45:36,804 DEBUG [RegionOpenAndInitThread-TestTable.testRecovery-1] regionserver.HRegion(798): Instantiated TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:36,805 DEBUG [RegionOpenAndInitThread-TestTable.testRecovery-1] regionserver.HRegion(1560): Closing TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.: disabling compactions & flushes 2017-10-03 21:45:36,805 DEBUG [RegionOpenAndInitThread-TestTable.testRecovery-1] regionserver.HRegion(1599): Updates disabled for region TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:36,805 INFO [RegionOpenAndInitThread-TestTable.testRecovery-1] regionserver.HRegion(1712): Closed TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:36,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=27 2017-10-03 21:45:36,919 DEBUG [ProcExecWrkr-13] hbase.MetaTableAccessor(1417): Put{"totalColumns":1,"row":"TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":9223372036854775807}]}} 2017-10-03 21:45:36,928 INFO [ProcExecWrkr-13] hbase.MetaTableAccessor(1614): Added 1 2017-10-03 21:45:36,992 DEBUG [ProcExecWrkr-13] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testRecovery","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067136991}]}} 2017-10-03 21:45:36,997 INFO [ProcExecWrkr-13] hbase.MetaTableAccessor(1770): Updated table TestTable.testRecovery state to ENABLING in META 2017-10-03 21:45:36,998 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=28, ppid=27, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a}] 2017-10-03 21:45:37,047 INFO [ProcExecWrkr-13] procedure.MasterProcedureScheduler(814): pid=28, ppid=27, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a TestTable.testRecovery TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:37,051 INFO [ProcExecWrkr-13] assignment.AssignProcedure(204): Start pid=28, ppid=27, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2017-10-03 21:45:37,204 INFO [ProcExecWrkr-14] assignment.RegionStateStore(186): pid=28 updating hbase:meta row=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a., regionState=OPENING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:37,222 INFO [ProcExecWrkr-14] assignment.RegionTransitionProcedure(213): Dispatch pid=28, ppid=27, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:37,373 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:45:37,375 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1916): Open TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:37,381 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@70efb718] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:41361 to delete [blk_1073741843_1019, blk_1073741844_1020] 2017-10-03 21:45:37,384 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(6654): Opening region: {ENCODED => e8d466bc47137a0d2d7b3119106dc88a, NAME => 'TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.', STARTKEY => '', ENDKEY => ''} 2017-10-03 21:45:37,385 INFO [RS_OPEN_REGION-cn012:37573-1] coprocessor.CoprocessorHost(164): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2017-10-03 21:45:37,385 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table TestTable.testRecovery e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:45:37,385 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(798): Instantiated TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:37,392 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/a 2017-10-03 21:45:37,392 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/a 2017-10-03 21:45:37,392 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:45:37,393 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:45:37,393 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] hfile.CacheConfig(258): Created cacheConfig for a: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:45:37,393 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:45:37,399 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/b 2017-10-03 21:45:37,399 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/b 2017-10-03 21:45:37,399 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:45:37,399 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:45:37,400 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] hfile.CacheConfig(258): Created cacheConfig for b: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:45:37,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=27 2017-10-03 21:45:37,400 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:45:37,407 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/c 2017-10-03 21:45:37,407 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/c 2017-10-03 21:45:37,407 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:45:37,407 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:45:37,407 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] hfile.CacheConfig(258): Created cacheConfig for c: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:45:37,408 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:45:37,409 INFO [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(1030): Setting FlushNonSloppyStoresFirstPolicy for the region=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:37,410 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(4121): Found 0 recovered edits file(s) under hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:45:37,413 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table TestTable.testRecovery, use config (44739242) instead 2017-10-03 21:45:37,417 DEBUG [RS_OPEN_REGION-cn012:37573-1] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2017-10-03 21:45:37,417 INFO [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(957): Onlined e8d466bc47137a0d2d7b3119106dc88a; next sequenceid=2 2017-10-03 21:45:37,419 INFO [PostOpenDeployTasks:e8d466bc47137a0d2d7b3119106dc88a] regionserver.HRegionServer(2170): Post open deploy tasks for TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:37,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=28, ppid=27, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:37,422 INFO [ProcExecWrkr-16] assignment.RegionStateStore(186): pid=28 updating hbase:meta row=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a., regionState=OPEN, openSeqNum=2, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:37,422 DEBUG [PostOpenDeployTasks:e8d466bc47137a0d2d7b3119106dc88a] regionserver.HRegionServer(2197): Finished post open deploy task for TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:37,423 DEBUG [RS_OPEN_REGION-cn012:37573-1] handler.OpenRegionHandler(127): Opened TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. on cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:37,519 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=28, ppid=27, state=SUCCESS; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:45:37,587 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=27, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=TestTable.testRecovery; resume parent processing. 2017-10-03 21:45:37,588 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1246): Finished pid=28, ppid=27, state=SUCCESS; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a in 429msec 2017-10-03 21:45:37,588 DEBUG [ProcExecWrkr-17] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testRecovery","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067137588}]}} 2017-10-03 21:45:37,593 INFO [ProcExecWrkr-17] hbase.MetaTableAccessor(1770): Updated table TestTable.testRecovery state to ENABLED in META 2017-10-03 21:45:37,776 INFO [ProcExecWrkr-17] procedure2.ProcedureExecutor(1246): Finished pid=27, state=SUCCESS; CreateTableProcedure table=TestTable.testRecovery in 1.6580sec 2017-10-03 21:45:38,379 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:45:38,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=27 2017-10-03 21:45:38,404 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: CREATE, Table Name: default:TestTable.testRecovery completed 2017-10-03 21:45:38,405 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:45:38,413 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:45:38,442 INFO [Time-limited test] client.ConnectionUtils(123): regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0 server-side Connection retries=350 2017-10-03 21:45:38,442 INFO [Time-limited test] ipc.RpcExecutor(145): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2017-10-03 21:45:38,443 INFO [Time-limited test] ipc.RpcExecutor(145): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2017-10-03 21:45:38,443 INFO [Time-limited test] ipc.RpcExecutor(145): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2017-10-03 21:45:38,443 INFO [Time-limited test] ipc.RpcServerFactory(59): Creating org.apache.hadoop.hbase.ipc.SimpleRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2017-10-03 21:45:38,443 INFO [Time-limited test] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2017-10-03 21:45:38,444 INFO [Time-limited test] ipc.SimpleRpcServer$Listener(141): regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0: started 3 reader(s) listening on port=43156 2017-10-03 21:45:38,445 INFO [Time-limited test] hfile.CacheConfig(283): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:45:38,445 INFO [Time-limited test] hfile.CacheConfig(283): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:45:38,447 INFO [Time-limited test] fs.HFileSystem(352): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2017-10-03 21:45:38,450 INFO [Time-limited test] fs.HFileSystem(352): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2017-10-03 21:45:38,452 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(122): Process identifier=regionserver:43156 connecting to ZooKeeper ensemble=localhost:57257 2017-10-03 21:45:38,490 DEBUG [Time-limited test-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:431560x0, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2017-10-03 21:45:38,491 DEBUG [Time-limited test-EventThread] zookeeper.ZooKeeperWatcher(528): regionserver:43156-0x15ee432cd9b000b connected 2017-10-03 21:45:38,491 DEBUG [Time-limited test] zookeeper.ZKUtil(362): regionserver:43156-0x15ee432cd9b000b, quorum=localhost:57257, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2017-10-03 21:45:38,492 DEBUG [Time-limited test] zookeeper.ZKUtil(362): regionserver:43156-0x15ee432cd9b000b, quorum=localhost:57257, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2017-10-03 21:45:38,493 DEBUG [RpcServer.responder] ipc.SimpleRpcServerResponder(57): RpcServer.responder: starting 2017-10-03 21:45:38,493 INFO [Listener,port=43156] ipc.SimpleRpcServer$Listener(222): Listener,port=43156: starting 2017-10-03 21:45:38,493 DEBUG [Time-limited test] ipc.RpcExecutor(208): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=43156 2017-10-03 21:45:38,495 DEBUG [Time-limited test] ipc.RpcExecutor(208): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=43156 2017-10-03 21:45:38,496 DEBUG [Time-limited test] ipc.RpcExecutor(208): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=43156 2017-10-03 21:45:38,499 INFO [RS:2;cn012:43156] zookeeper.RecoverableZooKeeper(122): Process identifier=hconnection-0x6ab92736 connecting to ZooKeeper ensemble=localhost:57257 2017-10-03 21:45:38,561 DEBUG [RS:2;cn012:43156-EventThread] zookeeper.ZooKeeperWatcher(465): hconnection-0x6ab927360x0, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2017-10-03 21:45:38,562 DEBUG [RS:2;cn012:43156-EventThread] zookeeper.ZooKeeperWatcher(528): hconnection-0x6ab92736-0x15ee432cd9b000c connected 2017-10-03 21:45:38,563 DEBUG [RS:2;cn012:43156] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60414c46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2017-10-03 21:45:38,573 INFO [RS:2;cn012:43156] regionserver.HRegionServer(883): ClusterId : dddea9d5-dadc-42e2-817a-4e2f6ef8dc92 2017-10-03 21:45:38,573 DEBUG [RS:2;cn012:43156] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2017-10-03 21:45:38,619 DEBUG [RS:2;cn012:43156] zookeeper.RecoverableZooKeeper(675): Node /hbase/flush-table-proc/acquired already exists 2017-10-03 21:45:38,621 DEBUG [RS:2;cn012:43156] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2017-10-03 21:45:38,621 DEBUG [RS:2;cn012:43156] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2017-10-03 21:45:38,641 DEBUG [RS:2;cn012:43156] zookeeper.RecoverableZooKeeper(675): Node /hbase/online-snapshot/acquired already exists 2017-10-03 21:45:38,642 DEBUG [RS:2;cn012:43156] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2017-10-03 21:45:38,642 INFO [RS:2;cn012:43156] regionserver.MemStoreFlusher(108): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2017-10-03 21:45:38,643 INFO [RS:2;cn012:43156] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2017-10-03 21:45:38,643 INFO [RS:2;cn012:43156] regionserver.HRegionServer$CompactionChecker(1762): CompactionChecker runs every 1sec 2017-10-03 21:45:38,643 DEBUG [RS:2;cn012:43156] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e65020d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=cn012.l42scl.hortonworks.com/172.18.128.12:0 2017-10-03 21:45:38,644 DEBUG [RS:2;cn012:43156] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:RS:2;cn012:43156 2017-10-03 21:45:38,644 INFO [RS:2;cn012:43156] regionserver.RegionServerCoprocessorHost(63): System coprocessor loading is enabled 2017-10-03 21:45:38,644 INFO [RS:2;cn012:43156] regionserver.RegionServerCoprocessorHost(64): Table coprocessor loading is enabled 2017-10-03 21:45:38,644 INFO [RS:2;cn012:43156] regionserver.HRegionServer(2597): reportForDuty to master=cn012.l42scl.hortonworks.com,35273,1507067089134 with port=43156, startcode=1507067138439 2017-10-03 21:45:38,645 DEBUG [RS:2;cn012:43156] ipc.NettyRpcConnection(255): Connecting to cn012.l42scl.hortonworks.com/172.18.128.12:35273 2017-10-03 21:45:38,649 INFO [Reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=35273] ipc.ServerRpcConnection(528): Auth successful for hbase.hfs.2 (auth:SIMPLE) 2017-10-03 21:45:38,649 INFO [Reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=35273] ipc.ServerRpcConnection(558): Connection from 172.18.128.12 port: 60591 with version info: version: "2.0.0-alpha3" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/2-hbase" revision: "d3a817f212f5d012f3cc7c6277314b682a907b0b" user: "hbase" date: "Tue Oct 3 21:04:59 UTC 2017" src_checksum: "0a7d6891d3586c77eaed645726da640b" version_major: 2 version_minor: 0 2017-10-03 21:45:38,650 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.ServerManager(429): Registering server=cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:38,651 DEBUG [RS:2;cn012:43156] regionserver.HRegionServer(1567): Config from master: hbase.rootdir=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0 2017-10-03 21:45:38,651 DEBUG [RS:2;cn012:43156] regionserver.HRegionServer(1567): Config from master: fs.defaultFS=hdfs://localhost:35744 2017-10-03 21:45:38,651 DEBUG [RS:2;cn012:43156] regionserver.HRegionServer(1567): Config from master: hbase.master.info.port=-1 2017-10-03 21:45:38,707 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2017-10-03 21:45:38,707 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2017-10-03 21:45:38,708 DEBUG [RS:2;cn012:43156] zookeeper.ZKUtil(362): regionserver:43156-0x15ee432cd9b000b, quorum=localhost:57257, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:38,708 DEBUG [main-EventThread] zookeeper.ZKUtil(362): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:38,708 WARN [RS:2;cn012:43156] hbase.ZNodeClearer(61): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2017-10-03 21:45:38,708 DEBUG [main-EventThread] zookeeper.ZKUtil(362): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:38,709 DEBUG [RS:2;cn012:43156] regionserver.HRegionServer(1866): logDir=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/WALs/cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:38,709 DEBUG [main-EventThread] zookeeper.ZKUtil(362): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:38,709 DEBUG [main-EventThread] zookeeper.ZKUtil(362): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:38,711 INFO [RS:2;cn012:43156] wal.WALFactory(143): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.FSHLogProvider 2017-10-03 21:45:38,769 INFO [RS:2;cn012:43156] zookeeper.RecoverableZooKeeper(122): Process identifier=hconnection-0x412218a0 connecting to ZooKeeper ensemble=localhost:57257 2017-10-03 21:45:38,807 DEBUG [RS:2;cn012:43156-EventThread] zookeeper.ZooKeeperWatcher(465): hconnection-0x412218a00x0, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2017-10-03 21:45:38,808 DEBUG [RS:2;cn012:43156-EventThread] zookeeper.ZooKeeperWatcher(528): hconnection-0x412218a0-0x15ee432cd9b000d connected 2017-10-03 21:45:38,809 DEBUG [RS:2;cn012:43156] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ff91ebb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2017-10-03 21:45:38,809 DEBUG [RS:2;cn012:43156] regionserver.Replication(152): ReplicationStatisticsThread 300 2017-10-03 21:45:38,809 INFO [RS:2;cn012:43156] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2017-10-03 21:45:38,827 DEBUG [RS:2;cn012:43156] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-cn012:43156, corePoolSize=3, maxPoolSize=3 2017-10-03 21:45:38,828 DEBUG [RS:2;cn012:43156] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-cn012:43156, corePoolSize=1, maxPoolSize=1 2017-10-03 21:45:38,828 DEBUG [RS:2;cn012:43156] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-cn012:43156, corePoolSize=3, maxPoolSize=3 2017-10-03 21:45:38,828 DEBUG [RS:2;cn012:43156] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-cn012:43156, corePoolSize=3, maxPoolSize=3 2017-10-03 21:45:38,828 DEBUG [RS:2;cn012:43156] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-cn012:43156, corePoolSize=1, maxPoolSize=1 2017-10-03 21:45:38,828 DEBUG [RS:2;cn012:43156] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-cn012:43156, corePoolSize=2, maxPoolSize=2 2017-10-03 21:45:38,828 DEBUG [RS:2;cn012:43156] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-cn012:43156, corePoolSize=10, maxPoolSize=10 2017-10-03 21:45:38,829 DEBUG [RS:2;cn012:43156] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-cn012:43156, corePoolSize=3, maxPoolSize=3 2017-10-03 21:45:38,831 DEBUG [ReplicationExecutor-0] zookeeper.ZKUtil(362): regionserver:43156-0x15ee432cd9b000b, quorum=localhost:57257, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:38,831 DEBUG [ReplicationExecutor-0] zookeeper.ZKUtil(362): regionserver:43156-0x15ee432cd9b000b, quorum=localhost:57257, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:38,832 INFO [ReplicationExecutor-0] regionserver.ReplicationSourceManager$AdoptAbandonedQueuesWorker(782): Current list of replicators: [cn012.l42scl.hortonworks.com,40892,1507067106201, cn012.l42scl.hortonworks.com,43156,1507067138439, cn012.l42scl.hortonworks.com,37573,1507067090360] other RSs: [cn012.l42scl.hortonworks.com,43156,1507067138439, cn012.l42scl.hortonworks.com,37573,1507067090360] 2017-10-03 21:45:38,869 INFO [RS:2;cn012:43156] regionserver.HeapMemoryManager(212): Starting HeapMemoryTuner chore. 2017-10-03 21:45:38,869 INFO [SplitLogWorker-cn012:43156] regionserver.SplitLogWorker(134): SplitLogWorker cn012.l42scl.hortonworks.com,43156,1507067138439 starting 2017-10-03 21:45:38,869 INFO [RS:2;cn012:43156] regionserver.HRegionServer(1601): Serving as cn012.l42scl.hortonworks.com,43156,1507067138439, RpcServer on cn012.l42scl.hortonworks.com/172.18.128.12:43156, sessionid=0x15ee432cd9b000b 2017-10-03 21:45:38,869 DEBUG [RS:2;cn012:43156] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2017-10-03 21:45:38,869 DEBUG [RS:2;cn012:43156] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:38,869 DEBUG [RS:2;cn012:43156] procedure.ZKProcedureMemberRpcs(350): Starting procedure member 'cn012.l42scl.hortonworks.com,43156,1507067138439' 2017-10-03 21:45:38,869 DEBUG [RS:2;cn012:43156] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2017-10-03 21:45:38,870 DEBUG [RS:2;cn012:43156] procedure.ZKProcedureMemberRpcs(150): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2017-10-03 21:45:38,871 DEBUG [RS:2;cn012:43156] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2017-10-03 21:45:38,871 DEBUG [RS:2;cn012:43156] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2017-10-03 21:45:38,871 DEBUG [RS:2;cn012:43156] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:38,871 DEBUG [RS:2;cn012:43156] procedure.ZKProcedureMemberRpcs(350): Starting procedure member 'cn012.l42scl.hortonworks.com,43156,1507067138439' 2017-10-03 21:45:38,871 DEBUG [RS:2;cn012:43156] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2017-10-03 21:45:38,871 DEBUG [RS:2;cn012:43156] procedure.ZKProcedureMemberRpcs(150): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2017-10-03 21:45:38,872 DEBUG [RS:2;cn012:43156] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2017-10-03 21:45:38,872 INFO [RS:2;cn012:43156] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2017-10-03 21:45:38,872 INFO [RS:2;cn012:43156] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2017-10-03 21:45:38,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.ServerManager(772): New admin connection to cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:38,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] ipc.NettyRpcConnection(255): Connecting to cn012.l42scl.hortonworks.com/172.18.128.12:43156 2017-10-03 21:45:38,881 INFO [Reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=43156] ipc.ServerRpcConnection(528): Auth successful for hbase (auth:SIMPLE) 2017-10-03 21:45:38,881 INFO [Reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=43156] ipc.ServerRpcConnection(558): Connection from 172.18.128.12 port: 43024 with version info: version: "2.0.0-alpha3" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/2-hbase" revision: "d3a817f212f5d012f3cc7c6277314b682a907b0b" user: "hbase" date: "Tue Oct 3 21:04:59 UTC 2017" src_checksum: "0a7d6891d3586c77eaed645726da640b" version_major: 2 version_minor: 0 2017-10-03 21:45:38,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=43156] regionserver.RSRpcServices(2037): Warming up Region TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:38,893 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=43156] wal.AbstractFSWAL(412): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=cn012.l42scl.hortonworks.com%2C43156%2C1507067138439, suffix=, logDir=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/WALs/cn012.l42scl.hortonworks.com,43156,1507067138439, archiveDir=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/oldWALs 2017-10-03 21:45:38,904 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=43156] wal.AbstractFSWAL(696): New WAL /user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/WALs/cn012.l42scl.hortonworks.com,43156,1507067138439/cn012.l42scl.hortonworks.com%2C43156%2C1507067138439.1507067138893 2017-10-03 21:45:38,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=43156] wal.AbstractFSWAL(762): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41361,DS-38a5a5c8-5f80-4892-8855-c1003ac8c597,DISK]] 2017-10-03 21:45:38,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=43156] regionserver.HRegion(6721): HRegion.Warming up region: {ENCODED => e8d466bc47137a0d2d7b3119106dc88a, NAME => 'TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.', STARTKEY => '', ENDKEY => ''} 2017-10-03 21:45:38,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=43156] regionserver.HRegion(798): Instantiated TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:38,910 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/a 2017-10-03 21:45:38,910 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/a 2017-10-03 21:45:38,910 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:45:38,910 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:45:38,911 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] hfile.CacheConfig(258): Created cacheConfig for a: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:45:38,911 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:45:38,913 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/b 2017-10-03 21:45:38,913 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/b 2017-10-03 21:45:38,913 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:45:38,914 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:45:38,914 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] hfile.CacheConfig(258): Created cacheConfig for b: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:45:38,914 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:45:38,916 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/c 2017-10-03 21:45:38,916 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/c 2017-10-03 21:45:38,916 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:45:38,916 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:45:38,917 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] hfile.CacheConfig(258): Created cacheConfig for c: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:45:38,917 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:45:38,919 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=43156] regionserver.HRegion(1030): Setting FlushNonSloppyStoresFirstPolicy for the region=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:38,920 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster(1708): Client=hbase//172.18.128.12 move hri=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a., source=cn012.l42scl.hortonworks.com,37573,1507067090360, destination=cn012.l42scl.hortonworks.com,43156,1507067138439, running balancer 2017-10-03 21:45:39,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=29, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a., source=cn012.l42scl.hortonworks.com,37573,1507067090360, destination=cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:39,142 INFO [ProcExecWrkr-18] procedure.MasterProcedureScheduler(814): pid=29, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a., source=cn012.l42scl.hortonworks.com,37573,1507067090360, destination=cn012.l42scl.hortonworks.com,43156,1507067138439 TestTable.testRecovery TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:39,142 INFO [ProcExecWrkr-18] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a, server=cn012.l42scl.hortonworks.com,37573,1507067090360}] 2017-10-03 21:45:39,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure.ProcedureSyncWait(191): waitFor pid=29 2017-10-03 21:45:39,196 INFO [ProcExecWrkr-18] procedure.MasterProcedureScheduler(814): pid=30, ppid=29, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a, server=cn012.l42scl.hortonworks.com,37573,1507067090360 TestTable.testRecovery TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:39,196 INFO [ProcExecWrkr-18] assignment.RegionStateStore(186): pid=30 updating hbase:meta row=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a., regionState=CLOSING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:39,201 INFO [ProcExecWrkr-18] assignment.RegionTransitionProcedure(213): Dispatch pid=30, ppid=29, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:39,352 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:45:39,354 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1520): Close e8d466bc47137a0d2d7b3119106dc88a without moving 2017-10-03 21:45:39,354 DEBUG [RS_CLOSE_REGION-cn012:37573-1] handler.CloseRegionHandler(91): Processing close of TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:39,356 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(1560): Closing TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.: disabling compactions & flushes 2017-10-03 21:45:39,356 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(1599): Updates disabled for region TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:39,357 INFO [StoreCloserThread-TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.-1] regionserver.HStore(926): Closed a 2017-10-03 21:45:39,357 INFO [StoreCloserThread-TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.-1] regionserver.HStore(926): Closed b 2017-10-03 21:45:39,357 INFO [StoreCloserThread-TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.-1] regionserver.HStore(926): Closed c 2017-10-03 21:45:39,373 DEBUG [RS_CLOSE_REGION-cn012:37573-1] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/recovered.edits/4.seqid, newSeqId=4, maxSeqId=2 2017-10-03 21:45:39,374 DEBUG [RS_CLOSE_REGION-cn012:37573-1] coprocessor.CoprocessorHost(291): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2017-10-03 21:45:39,374 INFO [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(1712): Closed TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:39,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report CLOSED seqId=-1, pid=30, ppid=29, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:39,377 INFO [ProcExecWrkr-19] assignment.RegionStateStore(186): pid=30 updating hbase:meta row=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a., regionState=CLOSED 2017-10-03 21:45:39,378 DEBUG [RS_CLOSE_REGION-cn012:37573-1] handler.CloseRegionHandler(123): Closed TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:39,464 INFO [ProcExecWrkr-19] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=30, ppid=29, state=SUCCESS; UnassignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a, server=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:39,563 INFO [ProcExecWrkr-19] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=29, state=RUNNABLE:MOVE_REGION_ASSIGN; MoveRegionProcedure hri=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a., source=cn012.l42scl.hortonworks.com,37573,1507067090360, destination=cn012.l42scl.hortonworks.com,43156,1507067138439; resume parent processing. 2017-10-03 21:45:39,563 INFO [ProcExecWrkr-21] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=31, ppid=29, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a, target=cn012.l42scl.hortonworks.com,43156,1507067138439}] 2017-10-03 21:45:39,563 INFO [ProcExecWrkr-19] procedure2.ProcedureExecutor(1246): Finished pid=30, ppid=29, state=SUCCESS; UnassignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a, server=cn012.l42scl.hortonworks.com,37573,1507067090360 in 242msec 2017-10-03 21:45:39,573 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2017-10-03 21:45:39,733 INFO [ProcExecWrkr-22] procedure.MasterProcedureScheduler(814): pid=31, ppid=29, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a, target=cn012.l42scl.hortonworks.com,43156,1507067138439 TestTable.testRecovery TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:39,736 INFO [ProcExecWrkr-22] assignment.AssignProcedure(204): Start pid=31, ppid=29, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a, target=cn012.l42scl.hortonworks.com,43156,1507067138439; rit=OFFLINE, location=cn012.l42scl.hortonworks.com,43156,1507067138439; forceNewPlan=false, retain=false 2017-10-03 21:45:39,889 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2017-10-03 21:45:39,890 INFO [ProcExecWrkr-23] assignment.RegionStateStore(186): pid=31 updating hbase:meta row=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a., regionState=OPENING, regionLocation=cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:39,895 INFO [ProcExecWrkr-23] assignment.RegionTransitionProcedure(213): Dispatch pid=31, ppid=29, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a, target=cn012.l42scl.hortonworks.com,43156,1507067138439; rit=OPENING, location=cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:40,046 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,43156,1507067138439 version=2097152 2017-10-03 21:45:40,048 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=43156] regionserver.RSRpcServices(1916): Open TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:40,054 DEBUG [RS_OPEN_REGION-cn012:43156-0] regionserver.HRegion(6654): Opening region: {ENCODED => e8d466bc47137a0d2d7b3119106dc88a, NAME => 'TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.', STARTKEY => '', ENDKEY => ''} 2017-10-03 21:45:40,055 INFO [RS_OPEN_REGION-cn012:43156-0] coprocessor.CoprocessorHost(164): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2017-10-03 21:45:40,055 DEBUG [RS_OPEN_REGION-cn012:43156-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table TestTable.testRecovery e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:45:40,055 DEBUG [RS_OPEN_REGION-cn012:43156-0] regionserver.HRegion(798): Instantiated TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:40,059 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/a 2017-10-03 21:45:40,060 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/a 2017-10-03 21:45:40,060 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:45:40,060 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:45:40,060 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] hfile.CacheConfig(258): Created cacheConfig for a: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:45:40,060 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:45:40,063 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/b 2017-10-03 21:45:40,063 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/b 2017-10-03 21:45:40,063 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:45:40,063 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:45:40,063 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] hfile.CacheConfig(258): Created cacheConfig for b: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:45:40,063 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:45:40,066 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/c 2017-10-03 21:45:40,066 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/c 2017-10-03 21:45:40,066 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:45:40,066 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:45:40,067 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] hfile.CacheConfig(258): Created cacheConfig for c: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:45:40,067 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:45:40,068 INFO [RS_OPEN_REGION-cn012:43156-0] regionserver.HRegion(1030): Setting FlushNonSloppyStoresFirstPolicy for the region=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:40,071 DEBUG [RS_OPEN_REGION-cn012:43156-0] regionserver.HRegion(4121): Found 0 recovered edits file(s) under hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:45:40,073 DEBUG [RS_OPEN_REGION-cn012:43156-0] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table TestTable.testRecovery, use config (44739242) instead 2017-10-03 21:45:40,078 DEBUG [RS_OPEN_REGION-cn012:43156-0] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/recovered.edits/5.seqid, newSeqId=5, maxSeqId=4 2017-10-03 21:45:40,080 INFO [RS_OPEN_REGION-cn012:43156-0] regionserver.HRegion(957): Onlined e8d466bc47137a0d2d7b3119106dc88a; next sequenceid=5 2017-10-03 21:45:40,082 INFO [PostOpenDeployTasks:e8d466bc47137a0d2d7b3119106dc88a] regionserver.HRegionServer(2170): Post open deploy tasks for TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:40,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=5, pid=31, ppid=29, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a, target=cn012.l42scl.hortonworks.com,43156,1507067138439; rit=OPENING, location=cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:40,086 INFO [ProcExecWrkr-1] assignment.RegionStateStore(186): pid=31 updating hbase:meta row=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a., regionState=OPEN, openSeqNum=5, regionLocation=cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:40,086 DEBUG [PostOpenDeployTasks:e8d466bc47137a0d2d7b3119106dc88a] regionserver.HRegionServer(2197): Finished post open deploy task for TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:40,087 DEBUG [RS_OPEN_REGION-cn012:43156-0] handler.OpenRegionHandler(127): Opened TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. on cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:40,154 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=31, ppid=29, state=SUCCESS; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a, target=cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:40,195 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=29, state=RUNNABLE; MoveRegionProcedure hri=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a., source=cn012.l42scl.hortonworks.com,37573,1507067090360, destination=cn012.l42scl.hortonworks.com,43156,1507067138439; resume parent processing. 2017-10-03 21:45:40,196 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1246): Finished pid=31, ppid=29, state=SUCCESS; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a, target=cn012.l42scl.hortonworks.com,43156,1507067138439 in 527msec 2017-10-03 21:45:40,237 INFO [ProcExecWrkr-24] procedure2.ProcedureExecutor(1246): Finished pid=29, state=SUCCESS; MoveRegionProcedure hri=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a., source=cn012.l42scl.hortonworks.com,37573,1507067090360, destination=cn012.l42scl.hortonworks.com,43156,1507067138439 in 1.2740sec 2017-10-03 21:45:40,248 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to cn012.l42scl.hortonworks.com/172.18.128.12:43156 2017-10-03 21:45:40,254 INFO [Reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=43156] ipc.ServerRpcConnection(528): Auth successful for hbase (auth:SIMPLE) 2017-10-03 21:45:40,254 INFO [Reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=43156] ipc.ServerRpcConnection(558): Connection from 172.18.128.12 port: 43130 with version info: version: "2.0.0-alpha3" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/2-hbase" revision: "d3a817f212f5d012f3cc7c6277314b682a907b0b" user: "hbase" date: "Tue Oct 3 21:04:59 UTC 2017" src_checksum: "0a7d6891d3586c77eaed645726da640b" version_major: 2 version_minor: 0 2017-10-03 21:45:40,264 INFO [Time-limited test] hbase.MiniHBaseCluster(245): Killing cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:40,264 FATAL [Time-limited test] regionserver.HRegionServer(2357): ***** ABORTING region server cn012.l42scl.hortonworks.com,43156,1507067138439: Simulated kill ***** 2017-10-03 21:45:40,264 FATAL [Time-limited test] regionserver.HRegionServer(2363): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver, org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2017-10-03 21:45:40,276 INFO [Time-limited test] regionserver.HRegionServer(2367): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 2315780096, "init" : 2109734912, "max" : 2609905664, "used" : 568956096 }, "NonHeapMemoryUsage" : { "committed" : 111804416, "init" : 2555904, "max" : -1, "used" : 110347968 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "cn012.l42scl.hortonworks.com", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 4, "numActiveHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 34841, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 145, "TotalCallTime_min" : 1, "TotalCallTime_max" : 6, "TotalCallTime_mean" : 3, "TotalCallTime_25th_percentile" : 2, "TotalCallTime_median" : 2, "TotalCallTime_75th_percentile" : 6, "TotalCallTime_90th_percentile" : 6, "TotalCallTime_95th_percentile" : 6, "TotalCallTime_98th_percentile" : 6, "TotalCallTime_99th_percentile" : 6, "TotalCallTime_99.9th_percentile" : 6, "TotalCallTime_TimeRangeCount_0-1" : 1, "TotalCallTime_TimeRangeCount_1-3" : 4, "TotalCallTime_TimeRangeCount_3-10" : 2, "TotalCallTime_TimeRangeCount_600000-inf" : 138, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 145, "ResponseSize_min" : 2, "ResponseSize_max" : 832, "ResponseSize_mean" : 132, "ResponseSize_25th_percentile" : 4, "ResponseSize_median" : 7, "ResponseSize_75th_percentile" : 63, "ResponseSize_90th_percentile" : 827, "ResponseSize_95th_percentile" : 829, "ResponseSize_98th_percentile" : 831, "ResponseSize_99th_percentile" : 831, "ResponseSize_99.9th_percentile" : 831, "ResponseSize_SizeRangeCount_0-10" : 5, "ResponseSize_SizeRangeCount_10-100" : 1, "ResponseSize_SizeRangeCount_100-1000" : 1, "ResponseSize_SizeRangeCount_100000000-inf" : 138, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 145, "ProcessCallTime_min" : 1, "ProcessCallTime_max" : 6, "ProcessCallTime_mean" : 3, "ProcessCallTime_25th_percentile" : 2, "ProcessCallTime_median" : 2, "ProcessCallTime_75th_percentile" : 6, "ProcessCallTime_90th_percentile" : 6, "ProcessCallTime_95th_percentile" : 6, "ProcessCallTime_98th_percentile" : 6, "ProcessCallTime_99th_percentile" : 6, "ProcessCallTime_99.9th_percentile" : 6, "ProcessCallTime_TimeRangeCount_0-1" : 1, "ProcessCallTime_TimeRangeCount_1-3" : 4, "ProcessCallTime_TimeRangeCount_3-10" : 2, "ProcessCallTime_TimeRangeCount_600000-inf" : 138, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 8, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 145, "RequestSize_min" : 94, "RequestSize_max" : 798, "RequestSize_mean" : 297, "RequestSize_25th_percentile" : 104, "RequestSize_median" : 191, "RequestSize_75th_percentile" : 563, "RequestSize_90th_percentile" : 796, "RequestSize_95th_percentile" : 797, "RequestSize_98th_percentile" : 797, "RequestSize_99th_percentile" : 797, "RequestSize_99.9th_percentile" : 797, "RequestSize_SizeRangeCount_10-100" : 1, "RequestSize_SizeRangeCount_100-1000" : 6, "RequestSize_SizeRangeCount_100000000-inf" : 138, "sentBytes" : 21029, "QueueCallTime_num_ops" : 145, "QueueCallTime_min" : 0, "QueueCallTime_max" : 0, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 0, "QueueCallTime_98th_percentile" : 0, "QueueCallTime_99th_percentile" : 0, "QueueCallTime_99.9th_percentile" : 0, "QueueCallTime_TimeRangeCount_0-1" : 7, "QueueCallTime_TimeRangeCount_600000-inf" : 138, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "cn012.l42scl.hortonworks.com", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:57257", "tag.serverName" : "cn012.l42scl.hortonworks.com,37573,1507067090360", "tag.clusterId" : "dddea9d5-dadc-42e2-817a-4e2f6ef8dc92", "tag.Context" : "regionserver", "tag.Hostname" : "cn012.l42scl.hortonworks.com", "regionCount" : 2, "storeCount" : 6, "hlogFileCount" : 2, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 11372, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1507067090360, "averageRegionSize" : 5686, "totalRequestCount" : 145, "totalRowActionRequestCount" : 126, "readRequestCount" : 58, "filteredReadRequestCount" : 7, "writeRequestCount" : 50, "rpcGetRequestCount" : 39, "rpcScanRequestCount" : 22, "rpcMultiRequestCount" : 57, "rpcMutateRequestCount" : 5, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043196664, "blockCacheCount" : 0, "blockCacheSize" : 765640, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : NaN, "l1CacheMissRatio" : NaN, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheCount" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "mobFileCacheHitPercent" : 0.0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 22, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_0-1" : 1, "ScanTime_TimeRangeCount_600000-inf" : 21, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 1, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Delete_TimeRangeCount_600000-inf" : 1, "Put_num_ops" : 3, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 3, "DeleteBatch_num_ops" : 9, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "DeleteBatch_TimeRangeCount_600000-inf" : 9, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 1, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 1, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 48, "PutBatch_min" : 2, "PutBatch_max" : 2, "PutBatch_mean" : 2, "PutBatch_25th_percentile" : 2, "PutBatch_median" : 2, "PutBatch_75th_percentile" : 2, "PutBatch_90th_percentile" : 2, "PutBatch_95th_percentile" : 2, "PutBatch_98th_percentile" : 2, "PutBatch_99th_percentile" : 2, "PutBatch_99.9th_percentile" : 2, "PutBatch_TimeRangeCount_1-3" : 2, "PutBatch_TimeRangeCount_600000-inf" : 46, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 39, "Get_min" : 1, "Get_max" : 1, "Get_mean" : 1, "Get_25th_percentile" : 1, "Get_median" : 1, "Get_75th_percentile" : 1, "Get_90th_percentile" : 1, "Get_95th_percentile" : 1, "Get_98th_percentile" : 1, "Get_99th_percentile" : 1, "Get_99.9th_percentile" : 1, "Get_TimeRangeCount_0-1" : 1, "Get_TimeRangeCount_600000-inf" : 38, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 14370, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 1, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 1, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 1, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CheckAndDelete_TimeRangeCount_600000-inf" : 1, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 22, "ScanSize_min" : 806, "ScanSize_max" : 806, "ScanSize_mean" : 806, "ScanSize_25th_percentile" : 806, "ScanSize_median" : 806, "ScanSize_75th_percentile" : 806, "ScanSize_90th_percentile" : 806, "ScanSize_95th_percentile" : 806, "ScanSize_98th_percentile" : 806, "ScanSize_99th_percentile" : 806, "ScanSize_99.9th_percentile" : 806, "ScanSize_SizeRangeCount_100-1000" : 1, "ScanSize_SizeRangeCount_100000000-inf" : 21, "slowGetCount" : 0, "flushedMemstoreBytes" : 102, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 1, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 1, "Bulkload_count" : 1, "Bulkload_mean_rate" : 0.020675701971460927, "Bulkload_1min_rate" : 0.015991117074135343, "Bulkload_5min_rate" : 0.0033057092356765017, "Bulkload_15min_rate" : 0.0011080303990206543, "Bulkload_num_ops" : 1, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2017-10-03 21:45:40,278 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(468): Region server cn012.l42scl.hortonworks.com,43156,1507067138439 reported a fatal error: ***** ABORTING region server cn012.l42scl.hortonworks.com,43156,1507067138439: Simulated kill ***** 2017-10-03 21:45:40,279 INFO [Time-limited test] regionserver.HRegionServer(2126): ***** STOPPING region server 'cn012.l42scl.hortonworks.com,43156,1507067138439' ***** 2017-10-03 21:45:40,282 INFO [Time-limited test] regionserver.HRegionServer(2140): STOPPED: Simulated kill 2017-10-03 21:45:40,282 INFO [RS:2;cn012:43156] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2017-10-03 21:45:40,283 INFO [RS:2;cn012:43156] regionserver.HeapMemoryManager(223): Stopping HeapMemoryTuner chore. 2017-10-03 21:45:40,283 INFO [SplitLogWorker-cn012:43156] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2017-10-03 21:45:40,284 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(275): MemStoreFlusher.0 exiting 2017-10-03 21:45:40,283 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(275): MemStoreFlusher.1 exiting 2017-10-03 21:45:40,284 INFO [SplitLogWorker-cn012:43156] regionserver.SplitLogWorker(155): SplitLogWorker cn012.l42scl.hortonworks.com,43156,1507067138439 exiting 2017-10-03 21:45:40,284 INFO [RS:2;cn012:43156] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager abruptly. 2017-10-03 21:45:40,284 INFO [RS:2;cn012:43156] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2017-10-03 21:45:40,285 DEBUG [RS:2;cn012:43156] zookeeper.MetaTableLocator(625): Stopping MetaTableLocator 2017-10-03 21:45:40,285 INFO [RS:2;cn012:43156] client.ConnectionImplementation(1247): Closing zookeeper sessionid=0x15ee432cd9b000c 2017-10-03 21:45:40,298 DEBUG [RS:2;cn012:43156] ipc.AbstractRpcClient(483): Stopping rpc client 2017-10-03 21:45:40,299 DEBUG [RS:2;cn012:43156] wal.FSHLog(434): Closing WAL writer in /user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/WALs/cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:40,302 INFO [IPC Server handler 2 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741848_1024{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|RBW]]} size 761 2017-10-03 21:45:40,704 DEBUG [RS:2;cn012:43156] ipc.AbstractRpcClient(483): Stopping rpc client 2017-10-03 21:45:40,704 INFO [RS:2;cn012:43156] regionserver.Leases(147): RS:2;cn012:43156 closing leases 2017-10-03 21:45:40,704 INFO [RS:2;cn012:43156] regionserver.Leases(150): RS:2;cn012:43156 closed leases 2017-10-03 21:45:40,705 INFO [RS:2;cn012:43156] ipc.SimpleRpcServer(440): Stopping server on 43156 2017-10-03 21:45:40,705 INFO [Listener,port=43156] ipc.SimpleRpcServer$Listener(267): Listener,port=43156: stopping 2017-10-03 21:45:40,706 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(157): RpcServer.responder: stopped 2017-10-03 21:45:40,708 INFO [RpcServer.responder] ipc.SimpleRpcServerResponder(61): RpcServer.responder: stopping 2017-10-03 21:45:40,829 INFO [cn012.l42scl.hortonworks.com,43156,1507067138439_Chore_1] hbase.ScheduledChore(181): Chore: MemstoreFlusherChore was stopped 2017-10-03 21:45:40,830 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker] regionserver.Leases(147): regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker closing leases 2017-10-03 21:45:40,830 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker] regionserver.Leases(150): regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker closed leases 2017-10-03 21:45:40,850 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:40,850 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:40,850 DEBUG [Time-limited test-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:43156-0x15ee432cd9b000b, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/cn012.l42scl.hortonworks.com,43156,1507067138439 2017-10-03 21:45:40,850 INFO [main-EventThread] zookeeper.RegionServerTracker(120): RegionServer ephemeral node deleted, processing expiration [cn012.l42scl.hortonworks.com,43156,1507067138439] 2017-10-03 21:45:40,851 DEBUG [Time-limited test-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:43156-0x15ee432cd9b000b, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2017-10-03 21:45:40,851 INFO [main-EventThread] master.ServerManager(593): Processing expiration of cn012.l42scl.hortonworks.com,43156,1507067138439 on cn012.l42scl.hortonworks.com,35273,1507067089134 2017-10-03 21:45:40,867 INFO [RS:2;cn012:43156] regionserver.HRegionServer(1225): stopping server cn012.l42scl.hortonworks.com,43156,1507067138439; zookeeper connection closed. 2017-10-03 21:45:40,867 INFO [RS:2;cn012:43156] regionserver.HRegionServer(1228): RS:2;cn012:43156 exiting 2017-10-03 21:45:40,868 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5f5a2580] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(191): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5f5a2580 2017-10-03 21:45:40,868 DEBUG [main-EventThread] zookeeper.ZKUtil(362): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:40,868 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(120): /hbase/rs/cn012.l42scl.hortonworks.com,43156,1507067138439 znode expired, triggering replicatorRemoved event 2017-10-03 21:45:40,869 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2017-10-03 21:45:40,869 DEBUG [main-EventThread] zookeeper.ZKUtil(362): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:41,012 DEBUG [main-EventThread] procedure2.ProcedureExecutor(868): Stored pid=32, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=cn012.l42scl.hortonworks.com,43156,1507067138439, splitWal=true, meta=false 2017-10-03 21:45:41,012 DEBUG [main-EventThread] assignment.AssignmentManager(1315): Added=cn012.l42scl.hortonworks.com,43156,1507067138439 to dead servers, submitted shutdown handler to be executed meta=false 2017-10-03 21:45:41,013 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2017-10-03 21:45:41,013 INFO [ProcExecWrkr-2] procedure.ServerCrashProcedure(118): Start pid=32, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=cn012.l42scl.hortonworks.com,43156,1507067138439, splitWal=true, meta=false 2017-10-03 21:45:41,014 DEBUG [main-EventThread] zookeeper.ZKUtil(362): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:41,131 DEBUG [ProcExecWrkr-3] procedure.ServerCrashProcedure(218): Splitting WALs pid=32, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=cn012.l42scl.hortonworks.com,43156,1507067138439, splitWal=true, meta=false 2017-10-03 21:45:41,134 DEBUG [ProcExecWrkr-3] master.MasterWalManager(262): Renamed region directory: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/WALs/cn012.l42scl.hortonworks.com,43156,1507067138439-splitting 2017-10-03 21:45:41,134 INFO [ProcExecWrkr-3] master.SplitLogManager(535): dead splitlog workers [cn012.l42scl.hortonworks.com,43156,1507067138439] 2017-10-03 21:45:41,135 INFO [ProcExecWrkr-3] master.SplitLogManager(252): Started splitting 1 logs in [hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/WALs/cn012.l42scl.hortonworks.com,43156,1507067138439-splitting] for [cn012.l42scl.hortonworks.com,43156,1507067138439] 2017-10-03 21:45:41,148 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2017-10-03 21:45:41,148 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination(426): put up splitlog task at znode /hbase/splitWAL/WALs%2Fcn012.l42scl.hortonworks.com%2C43156%2C1507067138439-splitting%2Fcn012.l42scl.hortonworks.com%252C43156%252C1507067138439.1507067138893 2017-10-03 21:45:41,149 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination(459): task not yet acquired /hbase/splitWAL/WALs%2Fcn012.l42scl.hortonworks.com%2C43156%2C1507067138439-splitting%2Fcn012.l42scl.hortonworks.com%252C43156%252C1507067138439.1507067138893 ver = 0 2017-10-03 21:45:41,157 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fcn012.l42scl.hortonworks.com%2C43156%2C1507067138439-splitting%2Fcn012.l42scl.hortonworks.com%252C43156%252C1507067138439.1507067138893 2017-10-03 21:45:41,157 INFO [SplitLogWorker-cn012:37573] coordination.ZkSplitLogWorkerCoordination(264): worker cn012.l42scl.hortonworks.com,37573,1507067090360 acquired task /hbase/splitWAL/WALs%2Fcn012.l42scl.hortonworks.com%2C43156%2C1507067138439-splitting%2Fcn012.l42scl.hortonworks.com%252C43156%252C1507067138439.1507067138893 2017-10-03 21:45:41,157 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(549): task /hbase/splitWAL/WALs%2Fcn012.l42scl.hortonworks.com%2C43156%2C1507067138439-splitting%2Fcn012.l42scl.hortonworks.com%252C43156%252C1507067138439.1507067138893 acquired by cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:41,178 INFO [RS_LOG_REPLAY_OPS-cn012:37573-1] wal.WALSplitter(291): Splitting WAL=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/WALs/cn012.l42scl.hortonworks.com,43156,1507067138439-splitting/cn012.l42scl.hortonworks.com%2C43156%2C1507067138439.1507067138893, length=761, distributedLogReplay=false 2017-10-03 21:45:41,181 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fcn012.l42scl.hortonworks.com%2C43156%2C1507067138439-splitting%2Fcn012.l42scl.hortonworks.com%252C43156%252C1507067138439.1507067138893 2017-10-03 21:45:41,181 INFO [RS_LOG_REPLAY_OPS-cn012:37573-1] util.FSHDFSUtils(195): Recover lease on dfs file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/WALs/cn012.l42scl.hortonworks.com,43156,1507067138439-splitting/cn012.l42scl.hortonworks.com%2C43156%2C1507067138439.1507067138893 2017-10-03 21:45:41,181 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fcn012.l42scl.hortonworks.com%2C43156%2C1507067138439-splitting%2Fcn012.l42scl.hortonworks.com%252C43156%252C1507067138439.1507067138893 2017-10-03 21:45:41,182 INFO [RS_LOG_REPLAY_OPS-cn012:37573-1] util.FSHDFSUtils(284): Recovered lease, attempt=0 on file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/WALs/cn012.l42scl.hortonworks.com,43156,1507067138439-splitting/cn012.l42scl.hortonworks.com%2C43156%2C1507067138439.1507067138893 after 1ms 2017-10-03 21:45:41,189 DEBUG [RS_LOG_REPLAY_OPS-cn012:37573-1] wal.WALSplitter(340): DLS Last flushed sequenceid for e8d466bc47137a0d2d7b3119106dc88a: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "a" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "b" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "c" sequence_id: 18446744073709551615 } 2017-10-03 21:45:41,190 DEBUG [RS_LOG_REPLAY_OPS-cn012:37573-1] wal.WALSplitter(393): Finishing writing output logs and closing down. 2017-10-03 21:45:41,190 DEBUG [RS_LOG_REPLAY_OPS-cn012:37573-1] wal.WALSplitter$OutputSink(1221): Waiting for split writer threads to finish 2017-10-03 21:45:41,195 DEBUG [RS_LOG_REPLAY_OPS-cn012:37573-1-Writer-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1547): Creating writer path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/recovered.edits/0000000000000000007-cn012.l42scl.hortonworks.com%2C43156%2C1507067138439.1507067138893.temp 2017-10-03 21:45:41,196 INFO [RS_LOG_REPLAY_OPS-cn012:37573-1] wal.WALSplitter$OutputSink(1245): 3 split writers finished; closing... 2017-10-03 21:45:41,203 INFO [IPC Server handler 5 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741849_1025{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|FINALIZED]]} size 0 2017-10-03 21:45:41,204 DEBUG [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink$2(1392): Closed wap hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/recovered.edits/0000000000000000007-cn012.l42scl.hortonworks.com%2C43156%2C1507067138439.1507067138893.temp (wrote 2 edits, skipped 0 edits in 6ms 2017-10-03 21:45:41,206 INFO [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink$2(1418): Rename hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/recovered.edits/0000000000000000007-cn012.l42scl.hortonworks.com%2C43156%2C1507067138439.1507067138893.temp to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/recovered.edits/0000000000000000008 2017-10-03 21:45:41,207 INFO [RS_LOG_REPLAY_OPS-cn012:37573-1] wal.WALSplitter(415): Processed 2 edits across 1 regions; edits skipped=1; log file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/WALs/cn012.l42scl.hortonworks.com,43156,1507067138439-splitting/cn012.l42scl.hortonworks.com%2C43156%2C1507067138439.1507067138893, length=761, corrupted=false, progress failed=false 2017-10-03 21:45:41,214 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fcn012.l42scl.hortonworks.com%2C43156%2C1507067138439-splitting%2Fcn012.l42scl.hortonworks.com%252C43156%252C1507067138439.1507067138893 2017-10-03 21:45:41,215 INFO [RS_LOG_REPLAY_OPS-cn012:37573-1] coordination.ZkSplitLogWorkerCoordination(597): successfully transitioned task /hbase/splitWAL/WALs%2Fcn012.l42scl.hortonworks.com%2C43156%2C1507067138439-splitting%2Fcn012.l42scl.hortonworks.com%252C43156%252C1507067138439.1507067138893 to final state DONE cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:41,214 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fcn012.l42scl.hortonworks.com%2C43156%2C1507067138439-splitting%2Fcn012.l42scl.hortonworks.com%252C43156%252C1507067138439.1507067138893 2017-10-03 21:45:41,215 INFO [RS_LOG_REPLAY_OPS-cn012:37573-1] handler.WALSplitterHandler(101): worker cn012.l42scl.hortonworks.com,37573,1507067090360 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@3a43b9b9 in 57ms 2017-10-03 21:45:41,216 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(467): task /hbase/splitWAL/WALs%2Fcn012.l42scl.hortonworks.com%2C43156%2C1507067138439-splitting%2Fcn012.l42scl.hortonworks.com%252C43156%252C1507067138439.1507067138893 entered state: DONE cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:41,220 INFO [main-EventThread] wal.WALSplitter(507): Archived processed log hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/WALs/cn012.l42scl.hortonworks.com,43156,1507067138439-splitting/cn012.l42scl.hortonworks.com%2C43156%2C1507067138439.1507067138893 to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/oldWALs/cn012.l42scl.hortonworks.com%2C43156%2C1507067138439.1507067138893 2017-10-03 21:45:41,220 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(510): Done splitting /hbase/splitWAL/WALs%2Fcn012.l42scl.hortonworks.com%2C43156%2C1507067138439-splitting%2Fcn012.l42scl.hortonworks.com%252C43156%252C1507067138439.1507067138893 2017-10-03 21:45:41,222 INFO [ProcExecWrkr-3] master.SplitLogManager(312): finished splitting (more than or equal to) 761 bytes in 1 log files in [hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/WALs/cn012.l42scl.hortonworks.com,43156,1507067138439-splitting] in 87ms 2017-10-03 21:45:41,223 DEBUG [ProcExecWrkr-3] procedure.ServerCrashProcedure(226): Done splitting WALs pid=32, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=cn012.l42scl.hortonworks.com,43156,1507067138439, splitWal=true, meta=false 2017-10-03 21:45:41,223 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fcn012.l42scl.hortonworks.com%2C43156%2C1507067138439-splitting%2Fcn012.l42scl.hortonworks.com%252C43156%252C1507067138439.1507067138893 2017-10-03 21:45:41,223 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fcn012.l42scl.hortonworks.com%2C43156%2C1507067138439-splitting%2Fcn012.l42scl.hortonworks.com%252C43156%252C1507067138439.1507067138893 2017-10-03 21:45:41,223 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination$DeleteAsyncCallback(1066): deleted /hbase/splitWAL/WALs%2Fcn012.l42scl.hortonworks.com%2C43156%2C1507067138439-splitting%2Fcn012.l42scl.hortonworks.com%252C43156%252C1507067138439.1507067138893 2017-10-03 21:45:41,223 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2017-10-03 21:45:41,271 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a}] 2017-10-03 21:45:41,283 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:45:41,287 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:45:41,287 INFO [Time-limited test] coprocessor.TestRegionObserverInterface(618): All regions assigned 2017-10-03 21:45:41,288 INFO [Time-limited test] client.HBaseAdmin$15(919): Started disable of TestTable.testRecovery 2017-10-03 21:45:41,289 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$11(2331): Client=hbase//172.18.128.12 disable TestTable.testRecovery 2017-10-03 21:45:41,322 INFO [ProcExecWrkr-3] procedure.MasterProcedureScheduler(814): pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a TestTable.testRecovery TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:41,326 INFO [ProcExecWrkr-3] assignment.AssignProcedure(204): Start pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OFFLINE, location=cn012.l42scl.hortonworks.com,43156,1507067138439; forceNewPlan=false, retain=true 2017-10-03 21:45:41,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=34, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestTable.testRecovery 2017-10-03 21:45:41,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=34 2017-10-03 21:45:41,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=34 2017-10-03 21:45:41,661 DEBUG [ProcExecWrkr-8] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testRecovery","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067141660}]}} 2017-10-03 21:45:41,665 INFO [ProcExecWrkr-8] hbase.MetaTableAccessor(1770): Updated table TestTable.testRecovery state to DISABLING in META 2017-10-03 21:45:41,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=34 2017-10-03 21:45:41,879 DEBUG [ProcExecWrkr-8] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testRecovery","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067141879}]}} 2017-10-03 21:45:41,884 INFO [ProcExecWrkr-8] hbase.MetaTableAccessor(1770): Updated table TestTable.testRecovery state to DISABLED in META 2017-10-03 21:45:41,884 INFO [ProcExecWrkr-8] procedure.DisableTableProcedure(285): Disabled table, TestTable.testRecovery, is completed. 2017-10-03 21:45:41,954 INFO [ProcExecWrkr-8] procedure2.ProcedureExecutor(1246): Finished pid=34, state=SUCCESS; DisableTableProcedure table=TestTable.testRecovery in 632msec 2017-10-03 21:45:41,955 INFO [ProcExecWrkr-8] assignment.RegionStateStore(186): pid=33 updating hbase:meta row=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a., regionState=OPENING 2017-10-03 21:45:41,961 INFO [ProcExecWrkr-8] assignment.RegionTransitionProcedure(213): Dispatch pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:41,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=34 2017-10-03 21:45:41,980 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DISABLE, Table Name: default:TestTable.testRecovery completed 2017-10-03 21:45:41,981 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$5(2109): Client=hbase//172.18.128.12 delete TestTable.testRecovery 2017-10-03 21:45:42,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=35, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestTable.testRecovery 2017-10-03 21:45:42,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:45:42,072 DEBUG [ProcExecWrkr-9] procedure.DeleteTableProcedure(99): Waiting for 'TestTable.testRecovery' regions in transition 2017-10-03 21:45:42,111 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:45:42,114 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1916): Open TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:42,120 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(6654): Opening region: {ENCODED => e8d466bc47137a0d2d7b3119106dc88a, NAME => 'TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.', STARTKEY => '', ENDKEY => ''} 2017-10-03 21:45:42,121 INFO [RS_OPEN_REGION-cn012:37573-2] coprocessor.CoprocessorHost(164): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2017-10-03 21:45:42,121 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table TestTable.testRecovery e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:45:42,121 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(798): Instantiated TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:42,123 DEBUG [ProcExecWrkr-9] procedure.ProcedureSyncWait(191): waitFor regions TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. in transition 2017-10-03 21:45:42,125 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/a 2017-10-03 21:45:42,126 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/a 2017-10-03 21:45:42,126 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:45:42,126 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:45:42,126 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] hfile.CacheConfig(258): Created cacheConfig for a: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:45:42,126 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:45:42,129 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/b 2017-10-03 21:45:42,129 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/b 2017-10-03 21:45:42,129 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:45:42,129 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:45:42,130 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] hfile.CacheConfig(258): Created cacheConfig for b: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:45:42,130 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:45:42,132 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/c 2017-10-03 21:45:42,132 DEBUG [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/c 2017-10-03 21:45:42,132 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:45:42,132 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:45:42,132 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] hfile.CacheConfig(258): Created cacheConfig for c: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:45:42,132 INFO [StoreOpener-e8d466bc47137a0d2d7b3119106dc88a-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:45:42,133 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(1030): Setting FlushNonSloppyStoresFirstPolicy for the region=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:42,137 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(4121): Found 1 recovered edits file(s) under hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:45:42,138 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(4218): Replaying edits from hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/recovered.edits/0000000000000000008 2017-10-03 21:45:42,150 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(4404): Applied 6, skipped 0, firstSequenceIdInLog=7, maxSequenceIdInLog=8, path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/recovered.edits/0000000000000000008 2017-10-03 21:45:42,150 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(2535): Flushing 3/3 column families, memstore=180 B 2017-10-03 21:45:42,150 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.store: a 2017-10-03 21:45:42,150 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:45:42,150 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.store: b 2017-10-03 21:45:42,150 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:45:42,151 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.store: c 2017-10-03 21:45:42,151 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:45:42,163 INFO [IPC Server handler 5 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741850_1026{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|RBW]]} size 4700 2017-10-03 21:45:42,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:45:42,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:45:42,566 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=8, memsize=60, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/.tmp/a/e585e11529bc43ddb0d4d7a99d4d4f1a 2017-10-03 21:45:42,590 INFO [IPC Server handler 0 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741851_1027{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|RBW]]} size 4700 2017-10-03 21:45:42,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:45:42,992 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=8, memsize=60, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/.tmp/b/5c646b59e80048099524bb2d88128dab 2017-10-03 21:45:43,019 INFO [IPC Server handler 9 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741852_1028{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|RBW]]} size 4700 2017-10-03 21:45:43,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:45:43,421 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=8, memsize=60, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/.tmp/c/162e81e1dc8d45b194bc4dcb3f20361f 2017-10-03 21:45:43,432 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/.tmp/a/e585e11529bc43ddb0d4d7a99d4d4f1a as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/a/e585e11529bc43ddb0d4d7a99d4d4f1a 2017-10-03 21:45:43,441 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/a/e585e11529bc43ddb0d4d7a99d4d4f1a, entries=2, sequenceid=8, filesize=4.6 K 2017-10-03 21:45:43,443 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/.tmp/b/5c646b59e80048099524bb2d88128dab as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/b/5c646b59e80048099524bb2d88128dab 2017-10-03 21:45:43,450 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/b/5c646b59e80048099524bb2d88128dab, entries=2, sequenceid=8, filesize=4.6 K 2017-10-03 21:45:43,452 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/.tmp/c/162e81e1dc8d45b194bc4dcb3f20361f as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/c/162e81e1dc8d45b194bc4dcb3f20361f 2017-10-03 21:45:43,458 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/c/162e81e1dc8d45b194bc4dcb3f20361f, entries=2, sequenceid=8, filesize=4.6 K 2017-10-03 21:45:43,459 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(2731): Finished memstore flush of ~180 B/180, currentsize=0 B/0 for region TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. in 1309ms, sequenceid=8, compaction requested=false; wal=null 2017-10-03 21:45:43,460 INFO [IPC Server handler 3 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741849_1025 127.0.0.1:41361 2017-10-03 21:45:43,460 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(4198): Deleted recovered.edits file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/recovered.edits/0000000000000000008 2017-10-03 21:45:43,461 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table TestTable.testRecovery, use config (44739242) instead 2017-10-03 21:45:43,464 DEBUG [RS_OPEN_REGION-cn012:37573-2] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/recovered.edits/9.seqid, newSeqId=9, maxSeqId=5 2017-10-03 21:45:43,465 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(957): Onlined e8d466bc47137a0d2d7b3119106dc88a; next sequenceid=9 2017-10-03 21:45:43,467 INFO [PostOpenDeployTasks:e8d466bc47137a0d2d7b3119106dc88a] regionserver.HRegionServer(2170): Post open deploy tasks for TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:43,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=9, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:43,469 DEBUG [PostOpenDeployTasks:e8d466bc47137a0d2d7b3119106dc88a] regionserver.HRegionServer(2197): Finished post open deploy task for TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:45:43,470 DEBUG [RS_OPEN_REGION-cn012:37573-2] handler.OpenRegionHandler(127): Opened TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. on cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:44,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:45:44,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:44,297 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.AssignProcedure(280): Skipping update of open seqnum with 0 because current seqnum=9 2017-10-03 21:45:44,878 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2017-10-03 21:45:44,903 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2017-10-03 21:45:45,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:46,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:45:46,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:46,367 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2017-10-03 21:45:46,382 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@70efb718] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:41361 to delete [blk_1073741849_1025] 2017-10-03 21:45:47,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:48,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:49,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:50,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:45:50,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:51,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:52,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:53,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:54,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:55,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:55,489 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data; isDirectory=true; modification_time=1507067103338; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:45:55,490 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default; isDirectory=true; modification_time=1507067135109; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:45:55,522 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks; isDirectory=true; modification_time=1507067103338; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest; isDirectory=true; modification_time=1507067135109; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE; isDirectory=true; modification_time=1507067129674; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:45:55,523 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e; isDirectory=true; modification_time=1507067103366; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:45:55,527 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/a; isDirectory=true; modification_time=1507067103348; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/b; isDirectory=true; modification_time=1507067103356; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/c; isDirectory=true; modification_time=1507067103362; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/recovered.edits; isDirectory=true; modification_time=1507067103369; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:45:55,537 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44; isDirectory=true; modification_time=1507067135125; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:45:55,539 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44/a; isDirectory=true; modification_time=1507067135117; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44/recovered.edits; isDirectory=true; modification_time=1507067135129; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:45:55,544 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE/65972395a311743485849c968cac9876; isDirectory=true; modification_time=1507067129682; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:45:55,544 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE/65972395a311743485849c968cac9876/recovered.edits; isDirectory=true; modification_time=1507067129686; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:45:56,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:56,566 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 14.4980sec 2017-10-03 21:45:57,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:58,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:45:59,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:00,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:46:00,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:01,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:01,566 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 19.4980sec 2017-10-03 21:46:02,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:03,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:04,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:05,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:06,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:06,566 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 24.4980sec 2017-10-03 21:46:07,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:08,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:08,387 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:46:09,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:10,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:46:10,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:10,834 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:46:11,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:11,566 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 29.4980sec 2017-10-03 21:46:11,577 INFO [ReplicationExecutor-0] regionserver.ReplicationSourceManager$NodeFailoverWorker(681): Not transferring queue since we are shutting down 2017-10-03 21:46:12,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:13,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:14,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:15,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:16,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:16,566 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 34.4980sec 2017-10-03 21:46:17,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:18,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:19,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:20,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:46:20,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:21,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:21,567 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 39.4990sec 2017-10-03 21:46:22,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:23,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:24,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:25,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:26,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:26,567 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 44.4990sec 2017-10-03 21:46:27,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:28,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:29,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:30,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:46:30,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:31,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:31,568 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 49.5000sec 2017-10-03 21:46:32,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:33,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:34,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:35,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:36,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:36,568 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 54.5000sec 2017-10-03 21:46:37,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:38,392 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:46:38,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:39,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:40,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:40,838 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:46:41,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:41,569 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 59.5010sec 2017-10-03 21:46:42,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:43,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:44,447 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:46:44,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:45,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:46,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:46,570 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 1mins, 4.502sec 2017-10-03 21:46:47,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:48,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:49,448 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:46:49,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:50,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:46:50,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:51,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:51,571 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 1mins, 9.503sec 2017-10-03 21:46:52,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:53,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:54,448 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:46:54,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:55,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:55,875 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data; isDirectory=true; modification_time=1507067103338; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:46:55,878 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default; isDirectory=true; modification_time=1507067135109; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:46:55,882 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks; isDirectory=true; modification_time=1507067103338; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest; isDirectory=true; modification_time=1507067135109; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE; isDirectory=true; modification_time=1507067129674; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:46:55,883 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e; isDirectory=true; modification_time=1507067103366; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:46:55,887 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/a; isDirectory=true; modification_time=1507067103348; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/b; isDirectory=true; modification_time=1507067103356; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/c; isDirectory=true; modification_time=1507067103362; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/recovered.edits; isDirectory=true; modification_time=1507067103369; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:46:55,895 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44; isDirectory=true; modification_time=1507067135125; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:46:55,897 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44/a; isDirectory=true; modification_time=1507067135117; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44/recovered.edits; isDirectory=true; modification_time=1507067135129; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:46:55,902 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE/65972395a311743485849c968cac9876; isDirectory=true; modification_time=1507067129682; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:46:55,903 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE/65972395a311743485849c968cac9876/recovered.edits; isDirectory=true; modification_time=1507067129686; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:46:56,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:56,571 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 1mins, 14.503sec 2017-10-03 21:46:57,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:58,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:46:59,449 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:46:59,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:00,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:01,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:01,571 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 1mins, 19.503sec 2017-10-03 21:47:02,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:03,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:04,451 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:47:04,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:05,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:06,399 INFO [cn012.l42scl.hortonworks.com,40892,1507067106201_Chore_1] hbase.ScheduledChore(181): Chore: MovedRegionsCleaner for region cn012.l42scl.hortonworks.com,40892,1507067106201 was stopped 2017-10-03 21:47:06,399 INFO [cn012.l42scl.hortonworks.com,40892,1507067106201_Chore_1] hbase.ScheduledChore(181): Chore: CompactedHFilesCleaner was stopped 2017-10-03 21:47:06,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:06,571 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 1mins, 24.503sec 2017-10-03 21:47:07,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:08,396 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:47:08,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:09,451 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:47:09,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:10,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:47:10,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:10,842 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:47:11,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:11,572 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 1mins, 29.504sec 2017-10-03 21:47:12,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:13,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:14,452 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:47:14,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:15,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:16,572 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 1mins, 34.504sec 2017-10-03 21:47:16,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:17,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:18,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:19,453 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:47:19,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:20,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:21,574 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 1mins, 39.506sec 2017-10-03 21:47:21,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:22,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:23,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:24,454 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:47:24,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:25,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:26,574 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 1mins, 44.506sec 2017-10-03 21:47:26,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:27,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:28,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:29,454 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:47:29,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:30,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:47:30,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:31,576 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 1mins, 49.508sec 2017-10-03 21:47:31,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:32,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:33,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:34,454 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:47:34,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:35,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:36,576 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 1mins, 54.508sec 2017-10-03 21:47:36,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:37,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:38,400 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:47:38,648 INFO [cn012.l42scl.hortonworks.com,43156,1507067138439_Chore_1] hbase.ScheduledChore(181): Chore: CompactedHFilesCleaner was stopped 2017-10-03 21:47:38,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:38,829 INFO [cn012.l42scl.hortonworks.com,43156,1507067138439_Chore_1] hbase.ScheduledChore(181): Chore: MovedRegionsCleaner for region cn012.l42scl.hortonworks.com,43156,1507067138439 was stopped 2017-10-03 21:47:39,454 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:47:39,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:40,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:40,846 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:47:41,576 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 1mins, 59.508sec 2017-10-03 21:47:41,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:42,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:43,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:44,455 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:47:44,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:45,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:46,576 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 2mins, 4.508sec 2017-10-03 21:47:46,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:47,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:48,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:49,457 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:47:49,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:50,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:47:50,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:51,576 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 2mins, 9.508sec 2017-10-03 21:47:51,616 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data; isDirectory=true; modification_time=1507067103338; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:47:51,618 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default; isDirectory=true; modification_time=1507067135109; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:47:51,623 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks; isDirectory=true; modification_time=1507067103338; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest; isDirectory=true; modification_time=1507067135109; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE; isDirectory=true; modification_time=1507067129674; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:47:51,624 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e; isDirectory=true; modification_time=1507067103366; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:47:51,629 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/a; isDirectory=true; modification_time=1507067103348; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/b; isDirectory=true; modification_time=1507067103356; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/c; isDirectory=true; modification_time=1507067103362; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/recovered.edits; isDirectory=true; modification_time=1507067103369; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:47:51,647 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44; isDirectory=true; modification_time=1507067135125; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:47:51,650 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44/a; isDirectory=true; modification_time=1507067135117; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44/recovered.edits; isDirectory=true; modification_time=1507067135129; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:47:51,658 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE/65972395a311743485849c968cac9876; isDirectory=true; modification_time=1507067129682; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:47:51,659 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE/65972395a311743485849c968cac9876/recovered.edits; isDirectory=true; modification_time=1507067129686; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:47:51,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:52,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:53,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:54,459 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:47:54,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:55,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:56,576 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 2mins, 14.508sec 2017-10-03 21:47:56,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:57,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:58,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:47:59,459 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:47:59,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:00,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:01,578 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 2mins, 19.509sec 2017-10-03 21:48:01,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:02,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:03,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:04,459 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:48:04,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:05,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:06,578 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 2mins, 24.51sec 2017-10-03 21:48:06,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:07,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:08,405 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:48:08,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:09,460 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:48:09,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:10,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:48:10,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:10,850 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:48:11,580 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 2mins, 29.512sec 2017-10-03 21:48:11,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:12,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:13,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:14,461 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:48:14,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:15,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:16,580 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 2mins, 34.512sec 2017-10-03 21:48:16,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:17,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:18,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:19,462 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:48:19,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:20,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:21,580 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 2mins, 39.512sec 2017-10-03 21:48:21,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:22,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:23,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:24,462 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:48:24,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:25,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:26,581 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 2mins, 44.512sec 2017-10-03 21:48:26,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:27,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:28,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:29,462 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:48:29,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:30,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:48:30,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:31,581 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 2mins, 49.513sec 2017-10-03 21:48:31,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:32,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:33,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:34,464 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:48:34,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:35,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:36,581 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 2mins, 54.513sec 2017-10-03 21:48:36,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:37,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:38,409 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:48:38,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:39,464 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:48:39,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:40,854 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:48:40,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:41,582 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 2mins, 59.514sec 2017-10-03 21:48:41,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:42,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:43,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:44,466 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:48:44,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:45,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:46,583 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 3mins, 4.514sec 2017-10-03 21:48:46,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:47,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:48,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:49,466 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:48:49,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:50,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:48:50,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:51,583 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 3mins, 9.515sec 2017-10-03 21:48:51,675 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data; isDirectory=true; modification_time=1507067103338; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:48:51,677 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default; isDirectory=true; modification_time=1507067135109; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:48:51,683 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks; isDirectory=true; modification_time=1507067103338; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest; isDirectory=true; modification_time=1507067135109; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE; isDirectory=true; modification_time=1507067129674; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:48:51,684 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e; isDirectory=true; modification_time=1507067103366; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:48:51,689 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/a; isDirectory=true; modification_time=1507067103348; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/b; isDirectory=true; modification_time=1507067103356; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/c; isDirectory=true; modification_time=1507067103362; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/recovered.edits; isDirectory=true; modification_time=1507067103369; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:48:51,702 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44; isDirectory=true; modification_time=1507067135125; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:48:51,704 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44/a; isDirectory=true; modification_time=1507067135117; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44/recovered.edits; isDirectory=true; modification_time=1507067135129; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:48:51,710 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE/65972395a311743485849c968cac9876; isDirectory=true; modification_time=1507067129682; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:48:51,711 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE/65972395a311743485849c968cac9876/recovered.edits; isDirectory=true; modification_time=1507067129686; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:48:51,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:52,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:53,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:54,467 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:48:54,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:55,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:56,584 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 3mins, 14.516sec 2017-10-03 21:48:56,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:57,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:58,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:48:59,468 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:48:59,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:00,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:01,585 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 3mins, 19.516sec 2017-10-03 21:49:01,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:02,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:03,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:04,468 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:49:04,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:05,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:06,585 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 3mins, 24.517sec 2017-10-03 21:49:06,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:07,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:08,413 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:49:08,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:09,470 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:49:09,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:10,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:49:10,858 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:49:10,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:11,586 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 3mins, 29.518sec 2017-10-03 21:49:11,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:13,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:14,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:14,470 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:49:15,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:16,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:16,588 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 3mins, 34.52sec 2017-10-03 21:49:17,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:18,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:19,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:19,471 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:49:20,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:21,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:21,588 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 3mins, 39.52sec 2017-10-03 21:49:22,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:23,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:24,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:24,471 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:49:25,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:26,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:26,589 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 3mins, 44.521sec 2017-10-03 21:49:27,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:28,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:29,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:29,473 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:49:30,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:30,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:49:31,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:31,589 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 3mins, 49.521sec 2017-10-03 21:49:32,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:33,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:34,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:34,473 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:49:35,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:36,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:36,590 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 3mins, 54.522sec 2017-10-03 21:49:37,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:38,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:38,418 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:49:39,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:39,473 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:49:40,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:40,862 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:49:41,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:41,590 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 3mins, 59.522sec 2017-10-03 21:49:42,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:43,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:44,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:44,475 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:49:45,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:46,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:46,591 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 4mins, 4.522sec 2017-10-03 21:49:47,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:48,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:49,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:49,475 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:49:50,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:50,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:49:50,391 DEBUG [cn012:35273.masterManager-MemStoreChunkPool Statistics] regionserver.ChunkCreator$MemStoreChunkPool$StatisticsThread(309): Stats: current pool size=3,created chunk count=12,reused chunk count=3,reuseRatio=20.00% 2017-10-03 21:49:51,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:51,591 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 4mins, 9.523sec 2017-10-03 21:49:52,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:53,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:53,977 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because table has an old edit so flush to free WALs after random delay 268436ms 2017-10-03 21:49:54,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:54,476 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:49:54,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because table has an old edit so flush to free WALs after random delay 25774ms 2017-10-03 21:49:55,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:55,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 169158ms 2017-10-03 21:49:56,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:56,592 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 4mins, 14.524sec 2017-10-03 21:49:56,813 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data; isDirectory=true; modification_time=1507067103338; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:49:56,814 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default; isDirectory=true; modification_time=1507067135109; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:49:56,819 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks; isDirectory=true; modification_time=1507067103338; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest; isDirectory=true; modification_time=1507067135109; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE; isDirectory=true; modification_time=1507067129674; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:49:56,820 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e; isDirectory=true; modification_time=1507067103366; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:49:56,825 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/a; isDirectory=true; modification_time=1507067103348; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/b; isDirectory=true; modification_time=1507067103356; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/c; isDirectory=true; modification_time=1507067103362; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/recovered.edits; isDirectory=true; modification_time=1507067103369; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:49:56,837 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44; isDirectory=true; modification_time=1507067135125; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:49:56,839 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44/a; isDirectory=true; modification_time=1507067135117; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44/recovered.edits; isDirectory=true; modification_time=1507067135129; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:49:56,843 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE/65972395a311743485849c968cac9876; isDirectory=true; modification_time=1507067129682; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:49:56,844 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE/65972395a311743485849c968cac9876/recovered.edits; isDirectory=true; modification_time=1507067129686; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:49:56,849 INFO [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] master.HMaster(1397): Not running balancer because 1 region(s) in transition: [rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a] 2017-10-03 21:49:56,852 DEBUG [hconnection-0x6d399eac-shared-pool8-t41] ipc.NettyRpcConnection(255): Connecting to cn012.l42scl.hortonworks.com/172.18.128.12:37573 2017-10-03 21:49:56,858 INFO [Reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37573] ipc.ServerRpcConnection(528): Auth successful for hbase (auth:SIMPLE) 2017-10-03 21:49:56,858 INFO [Reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37573] ipc.ServerRpcConnection(558): Connection from 172.18.128.12 port: 38012 with version info: version: "2.0.0-alpha3" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/2-hbase" revision: "d3a817f212f5d012f3cc7c6277314b682a907b0b" user: "hbase" date: "Tue Oct 3 21:04:59 UTC 2017" src_checksum: "0a7d6891d3586c77eaed645726da640b" version_major: 2 version_minor: 0 2017-10-03 21:49:56,865 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] master.HMaster(1522): Skipping normalization for table: hbase:meta, as it's either system table or doesn't have auto normalization turned on 2017-10-03 21:49:56,865 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] master.HMaster(1522): Skipping normalization for table: hbase:namespace, as it's either system table or doesn't have auto normalization turned on 2017-10-03 21:49:56,866 WARN [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] master.CatalogJanitor(121): CatalogJanitor is disabled! Enabled=true, maintenanceMode=false, am=org.apache.hadoop.hbase.master.assignment.AssignmentManager@62336ddb, failoverCleanupDone=true, hasRIT=true 2017-10-03 21:49:56,919 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 162386ms 2017-10-03 21:49:56,919 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 127992ms 2017-10-03 21:49:57,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:57,931 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 176999ms 2017-10-03 21:49:57,932 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 149626ms 2017-10-03 21:49:58,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:59,003 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 219698ms 2017-10-03 21:49:59,003 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 283087ms 2017-10-03 21:49:59,069 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2017-10-03 21:49:59,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:49:59,476 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:49:59,926 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 115158ms 2017-10-03 21:49:59,926 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 34136ms 2017-10-03 21:50:00,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:00,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 44780ms 2017-10-03 21:50:00,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 119156ms 2017-10-03 21:50:01,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:01,593 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 4mins, 19.524sec 2017-10-03 21:50:02,001 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 263765ms 2017-10-03 21:50:02,001 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 287741ms 2017-10-03 21:50:02,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:02,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 255996ms 2017-10-03 21:50:02,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 254805ms 2017-10-03 21:50:03,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:03,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 129747ms 2017-10-03 21:50:03,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 149958ms 2017-10-03 21:50:04,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:04,476 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:50:04,948 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 98517ms 2017-10-03 21:50:04,949 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 134964ms 2017-10-03 21:50:05,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:05,918 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 97989ms 2017-10-03 21:50:05,918 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 87078ms 2017-10-03 21:50:06,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:06,593 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 4mins, 24.525sec 2017-10-03 21:50:06,920 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 137810ms 2017-10-03 21:50:06,920 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 293774ms 2017-10-03 21:50:07,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:07,939 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 31466ms 2017-10-03 21:50:07,939 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 239314ms 2017-10-03 21:50:08,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:08,421 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:50:08,924 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 11675ms 2017-10-03 21:50:08,924 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 38265ms 2017-10-03 21:50:09,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:09,478 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:50:09,931 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 16832ms 2017-10-03 21:50:09,931 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 259898ms 2017-10-03 21:50:10,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:10,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:50:10,866 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:50:10,991 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 189232ms 2017-10-03 21:50:10,991 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 9926ms 2017-10-03 21:50:11,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:11,593 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 4mins, 29.525sec 2017-10-03 21:50:11,993 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 76733ms 2017-10-03 21:50:11,993 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 290579ms 2017-10-03 21:50:12,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:12,948 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 58037ms 2017-10-03 21:50:12,949 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 136421ms 2017-10-03 21:50:13,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:13,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 4011ms 2017-10-03 21:50:13,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 45167ms 2017-10-03 21:50:14,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:14,479 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:50:14,962 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 173786ms 2017-10-03 21:50:14,963 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 264936ms 2017-10-03 21:50:15,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:15,992 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 94594ms 2017-10-03 21:50:15,992 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 203021ms 2017-10-03 21:50:16,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:16,594 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 4mins, 34.525sec 2017-10-03 21:50:16,920 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 270726ms 2017-10-03 21:50:16,921 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 251808ms 2017-10-03 21:50:17,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:17,987 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 256227ms 2017-10-03 21:50:17,987 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 66029ms 2017-10-03 21:50:18,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:18,921 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 233024ms 2017-10-03 21:50:18,921 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 68612ms 2017-10-03 21:50:19,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:19,479 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:50:19,951 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 137519ms 2017-10-03 21:50:19,952 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 25244ms 2017-10-03 21:50:20,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:21,010 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 152249ms 2017-10-03 21:50:21,010 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 92720ms 2017-10-03 21:50:21,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:21,594 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 4mins, 39.526sec 2017-10-03 21:50:21,725 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(220): Evict completed pid=7, state=SUCCESS; CreateTableProcedure table=TestTable.testCheckAndDeleteHooks 2017-10-03 21:50:21,725 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(220): Evict completed pid=9, state=SUCCESS; DisableTableProcedure table=TestTable.testCheckAndDeleteHooks 2017-10-03 21:50:21,725 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(220): Evict completed pid=11, state=SUCCESS; DeleteTableProcedure table=TestTable.testCheckAndDeleteHooks 2017-10-03 21:50:21,725 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(220): Evict completed pid=12, state=SUCCESS; CreateTableProcedure table=SKIPPED_BY_PREWALRESTORE 2017-10-03 21:50:21,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 123810ms 2017-10-03 21:50:21,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 29901ms 2017-10-03 21:50:22,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:22,918 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 253081ms 2017-10-03 21:50:22,918 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 142761ms 2017-10-03 21:50:23,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:23,918 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 223824ms 2017-10-03 21:50:23,919 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 8777ms 2017-10-03 21:50:24,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:24,481 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:50:24,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 239652ms 2017-10-03 21:50:24,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 287262ms 2017-10-03 21:50:25,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:25,982 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 160218ms 2017-10-03 21:50:25,982 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 255677ms 2017-10-03 21:50:26,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:26,594 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 4mins, 44.526sec 2017-10-03 21:50:26,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 46697ms 2017-10-03 21:50:26,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 142283ms 2017-10-03 21:50:27,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:27,993 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 171276ms 2017-10-03 21:50:27,993 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 207122ms 2017-10-03 21:50:28,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:28,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 278183ms 2017-10-03 21:50:28,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 213367ms 2017-10-03 21:50:29,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:29,482 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:50:29,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 74422ms 2017-10-03 21:50:29,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 25338ms 2017-10-03 21:50:30,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:30,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=35 2017-10-03 21:50:30,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 150716ms 2017-10-03 21:50:30,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 93822ms 2017-10-03 21:50:31,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:31,595 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 4mins, 49.527sec 2017-10-03 21:50:31,946 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 251237ms 2017-10-03 21:50:31,946 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 228710ms 2017-10-03 21:50:32,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:33,008 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 116908ms 2017-10-03 21:50:33,008 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 164425ms 2017-10-03 21:50:33,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:33,938 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 271581ms 2017-10-03 21:50:33,938 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 271157ms 2017-10-03 21:50:34,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:34,482 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:50:35,009 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 101802ms 2017-10-03 21:50:35,010 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 38520ms 2017-10-03 21:50:35,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:35,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 237974ms 2017-10-03 21:50:35,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 207878ms 2017-10-03 21:50:36,092 INFO [main] hbase.ResourceChecker(172): after: coprocessor.TestRegionObserverInterface#testRecovery Thread=253 (was 279), OpenFileDescriptor=1226 (was 1145) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32000 (was 32000), SystemLoadAverage=432 (was 710), ProcessCount=347 (was 347), AvailableMemoryMB=48352 (was 47494) - AvailableMemoryMB LEAK? - 2017-10-03 21:50:36,092 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1226 is superior to 1024 2017-10-03 21:50:36,129 INFO [main] hbase.ResourceChecker(148): before: coprocessor.TestRegionObserverInterface#testHBase3583 Thread=253, OpenFileDescriptor=1226, MaxFileDescriptor=32000, SystemLoadAverage=432, ProcessCount=347, AvailableMemoryMB=48351 2017-10-03 21:50:36,129 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1226 is superior to 1024 2017-10-03 21:50:36,137 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$3(1748): Client=hbase//172.18.128.12 create 'testHBase3583', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2017-10-03 21:50:36,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=36, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testHBase3583 2017-10-03 21:50:36,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:36,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=36 2017-10-03 21:50:36,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=36 2017-10-03 21:50:36,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=36 2017-10-03 21:50:36,595 INFO [IPC Server handler 6 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741853_1029{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|RBW]]} size 1428 2017-10-03 21:50:36,596 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 4mins, 54.528sec 2017-10-03 21:50:36,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=36 2017-10-03 21:50:36,942 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 268422ms 2017-10-03 21:50:36,942 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 5697ms 2017-10-03 21:50:37,008 DEBUG [ProcExecWrkr-12] util.FSTableDescriptors(728): Wrote descriptor into: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3583/.tabledesc/.tableinfo.0000000001 2017-10-03 21:50:37,013 INFO [RegionOpenAndInitThread-testHBase3583-1] regionserver.HRegion(6477): creating HRegion testHBase3583 HTD == 'testHBase3583', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp Table name == testHBase3583 2017-10-03 21:50:37,032 INFO [IPC Server handler 3 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741854_1030{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|FINALIZED]]} size 0 2017-10-03 21:50:37,034 DEBUG [RegionOpenAndInitThread-testHBase3583-1] regionserver.HRegion(798): Instantiated testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. 2017-10-03 21:50:37,035 DEBUG [RegionOpenAndInitThread-testHBase3583-1] regionserver.HRegion(1560): Closing testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc.: disabling compactions & flushes 2017-10-03 21:50:37,035 DEBUG [RegionOpenAndInitThread-testHBase3583-1] regionserver.HRegion(1599): Updates disabled for region testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. 2017-10-03 21:50:37,035 INFO [RegionOpenAndInitThread-testHBase3583-1] regionserver.HRegion(1712): Closed testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. 2017-10-03 21:50:37,231 DEBUG [ProcExecWrkr-12] hbase.MetaTableAccessor(1417): Put{"totalColumns":1,"row":"testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc.","families":{"info":[{"qualifier":"regioninfo","vlen":47,"tag":[],"timestamp":9223372036854775807}]}} 2017-10-03 21:50:37,236 INFO [ProcExecWrkr-12] hbase.MetaTableAccessor(1614): Added 1 2017-10-03 21:50:37,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:37,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=36 2017-10-03 21:50:37,449 DEBUG [ProcExecWrkr-12] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"testHBase3583","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067437449}]}} 2017-10-03 21:50:37,454 INFO [ProcExecWrkr-12] hbase.MetaTableAccessor(1770): Updated table testHBase3583 state to ENABLING in META 2017-10-03 21:50:37,455 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testHBase3583, region=4d1729514f896c7adccdd0a78f0eb9bc}] 2017-10-03 21:50:37,646 INFO [ProcExecWrkr-12] procedure.MasterProcedureScheduler(814): pid=37, ppid=36, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testHBase3583, region=4d1729514f896c7adccdd0a78f0eb9bc testHBase3583 testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. 2017-10-03 21:50:37,649 INFO [ProcExecWrkr-12] assignment.AssignProcedure(204): Start pid=37, ppid=36, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testHBase3583, region=4d1729514f896c7adccdd0a78f0eb9bc; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2017-10-03 21:50:37,803 INFO [ProcExecWrkr-15] assignment.RegionStateStore(186): pid=37 updating hbase:meta row=testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc., regionState=OPENING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:37,812 DEBUG [MultiHConnection-shared--pool9-t28] ipc.NettyRpcConnection(255): Connecting to cn012.l42scl.hortonworks.com/172.18.128.12:37573 2017-10-03 21:50:37,822 INFO [Reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=37573] ipc.ServerRpcConnection(528): Auth successful for hbase (auth:SIMPLE) 2017-10-03 21:50:37,823 INFO [Reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=37573] ipc.ServerRpcConnection(558): Connection from 172.18.128.12 port: 39260 with version info: version: "2.0.0-alpha3" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/2-hbase" revision: "d3a817f212f5d012f3cc7c6277314b682a907b0b" user: "hbase" date: "Tue Oct 3 21:04:59 UTC 2017" src_checksum: "0a7d6891d3586c77eaed645726da640b" version_major: 2 version_minor: 0 2017-10-03 21:50:37,828 INFO [ProcExecWrkr-15] assignment.RegionTransitionProcedure(213): Dispatch pid=37, ppid=36, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testHBase3583, region=4d1729514f896c7adccdd0a78f0eb9bc; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:37,979 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:50:37,981 DEBUG [RSProcedureDispatcher-pool4-t16] ipc.NettyRpcConnection(255): Connecting to cn012.l42scl.hortonworks.com/172.18.128.12:37573 2017-10-03 21:50:37,984 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 202832ms 2017-10-03 21:50:37,985 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 99950ms 2017-10-03 21:50:37,988 INFO [Reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=37573] ipc.ServerRpcConnection(528): Auth successful for hbase (auth:SIMPLE) 2017-10-03 21:50:37,989 INFO [Reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=37573] ipc.ServerRpcConnection(558): Connection from 172.18.128.12 port: 39262 with version info: version: "2.0.0-alpha3" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/2-hbase" revision: "d3a817f212f5d012f3cc7c6277314b682a907b0b" user: "hbase" date: "Tue Oct 3 21:04:59 UTC 2017" src_checksum: "0a7d6891d3586c77eaed645726da640b" version_major: 2 version_minor: 0 2017-10-03 21:50:37,990 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1916): Open testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. 2017-10-03 21:50:38,002 DEBUG [RS_OPEN_REGION-cn012:37573-0] regionserver.HRegion(6654): Opening region: {ENCODED => 4d1729514f896c7adccdd0a78f0eb9bc, NAME => 'testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc.', STARTKEY => '', ENDKEY => ''} 2017-10-03 21:50:38,003 INFO [RS_OPEN_REGION-cn012:37573-0] coprocessor.CoprocessorHost(164): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2017-10-03 21:50:38,003 DEBUG [RS_OPEN_REGION-cn012:37573-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testHBase3583 4d1729514f896c7adccdd0a78f0eb9bc 2017-10-03 21:50:38,004 DEBUG [RS_OPEN_REGION-cn012:37573-0] regionserver.HRegion(798): Instantiated testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. 2017-10-03 21:50:38,010 DEBUG [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/a 2017-10-03 21:50:38,011 DEBUG [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/a 2017-10-03 21:50:38,011 INFO [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:50:38,011 INFO [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:50:38,012 INFO [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] hfile.CacheConfig(258): Created cacheConfig for a: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:50:38,012 INFO [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:50:38,017 DEBUG [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/b 2017-10-03 21:50:38,017 DEBUG [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/b 2017-10-03 21:50:38,018 INFO [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:50:38,018 INFO [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:50:38,018 INFO [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] hfile.CacheConfig(258): Created cacheConfig for b: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:50:38,018 INFO [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:50:38,022 DEBUG [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/c 2017-10-03 21:50:38,023 DEBUG [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/c 2017-10-03 21:50:38,023 INFO [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:50:38,023 INFO [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:50:38,023 INFO [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] hfile.CacheConfig(258): Created cacheConfig for c: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:50:38,023 INFO [StoreOpener-4d1729514f896c7adccdd0a78f0eb9bc-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:50:38,025 INFO [RS_OPEN_REGION-cn012:37573-0] regionserver.HRegion(1030): Setting FlushNonSloppyStoresFirstPolicy for the region=testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. 2017-10-03 21:50:38,026 DEBUG [RS_OPEN_REGION-cn012:37573-0] regionserver.HRegion(4121): Found 0 recovered edits file(s) under hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc 2017-10-03 21:50:38,028 DEBUG [RS_OPEN_REGION-cn012:37573-0] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table testHBase3583, use config (44739242) instead 2017-10-03 21:50:38,034 DEBUG [RS_OPEN_REGION-cn012:37573-0] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2017-10-03 21:50:38,034 INFO [RS_OPEN_REGION-cn012:37573-0] regionserver.HRegion(957): Onlined 4d1729514f896c7adccdd0a78f0eb9bc; next sequenceid=2 2017-10-03 21:50:38,037 INFO [PostOpenDeployTasks:4d1729514f896c7adccdd0a78f0eb9bc] regionserver.HRegionServer(2170): Post open deploy tasks for testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. 2017-10-03 21:50:38,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=37, ppid=36, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testHBase3583, region=4d1729514f896c7adccdd0a78f0eb9bc; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:38,040 DEBUG [PostOpenDeployTasks:4d1729514f896c7adccdd0a78f0eb9bc] regionserver.HRegionServer(2197): Finished post open deploy task for testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. 2017-10-03 21:50:38,040 INFO [ProcExecWrkr-13] assignment.RegionStateStore(186): pid=37 updating hbase:meta row=testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc., regionState=OPEN, openSeqNum=2, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:38,041 DEBUG [RS_OPEN_REGION-cn012:37573-0] handler.OpenRegionHandler(127): Opened testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. on cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:38,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:38,320 WARN [DataXceiver for client DFSClient_NONMAPREDUCE_-174551704_1 at /127.0.0.1:58812 [Receiving block BP-608549805-172.18.128.12-1507067084534:blk_1073741829_1005]] datanode.BlockReceiver(391): Slow flushOrSync took 390ms (threshold=300ms), isSync:true, flushTotalNanos=9163ns 2017-10-03 21:50:38,321 INFO [ProcExecWrkr-15] procedure2.ProcedureExecutor(1246): Finished pid=37, ppid=36, state=SUCCESS; AssignProcedure table=testHBase3583, region=4d1729514f896c7adccdd0a78f0eb9bc in 592msec 2017-10-03 21:50:38,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=36 2017-10-03 21:50:38,426 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:50:38,527 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=37, ppid=36, state=SUCCESS; AssignProcedure table=testHBase3583, region=4d1729514f896c7adccdd0a78f0eb9bc 2017-10-03 21:50:38,805 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=36, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testHBase3583; resume parent processing. 2017-10-03 21:50:38,806 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1246): Finished pid=37, ppid=36, state=SUCCESS; AssignProcedure table=testHBase3583, region=4d1729514f896c7adccdd0a78f0eb9bc in 592msec 2017-10-03 21:50:38,806 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"testHBase3583","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067438806}]}} 2017-10-03 21:50:38,812 INFO [ProcExecWrkr-14] hbase.MetaTableAccessor(1770): Updated table testHBase3583 state to ENABLED in META 2017-10-03 21:50:38,919 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 40115ms 2017-10-03 21:50:38,919 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 30162ms 2017-10-03 21:50:39,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:39,283 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1246): Finished pid=36, state=SUCCESS; CreateTableProcedure table=testHBase3583 in 2.8840sec 2017-10-03 21:50:39,482 WARN [ProcExecTimeout] assignment.AssignmentManager(1143): TODO Handle stuck in transition: rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360, table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:50:40,004 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 137461ms 2017-10-03 21:50:40,004 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 153253ms 2017-10-03 21:50:40,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:40,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=36 2017-10-03 21:50:40,356 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: CREATE, Table Name: default:testHBase3583 completed 2017-10-03 21:50:40,358 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:50:40,360 DEBUG [hconnection-0x61a91c9b-shared-pool13-t16] ipc.NettyRpcConnection(255): Connecting to cn012.l42scl.hortonworks.com/172.18.128.12:37573 2017-10-03 21:50:40,367 INFO [Reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37573] ipc.ServerRpcConnection(528): Auth successful for hbase (auth:SIMPLE) 2017-10-03 21:50:40,375 INFO [Reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37573] ipc.ServerRpcConnection(558): Connection from 172.18.128.12 port: 39274 with version info: version: "2.0.0-alpha3" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/2-hbase" revision: "d3a817f212f5d012f3cc7c6277314b682a907b0b" user: "hbase" date: "Tue Oct 3 21:04:59 UTC 2017" src_checksum: "0a7d6891d3586c77eaed645726da640b" version_major: 2 version_minor: 0 2017-10-03 21:50:40,381 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:50:40,382 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:50:40,384 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2017-10-03 21:50:40,390 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:50:40,404 INFO [Time-limited test] client.HBaseAdmin$15(919): Started disable of testHBase3583 2017-10-03 21:50:40,405 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$11(2331): Client=hbase//172.18.128.12 disable testHBase3583 2017-10-03 21:50:40,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=38, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testHBase3583 2017-10-03 21:50:40,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=38 2017-10-03 21:50:40,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=38 2017-10-03 21:50:40,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=38 2017-10-03 21:50:40,870 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:50:40,879 DEBUG [ProcExecWrkr-16] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"testHBase3583","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067440879}]}} 2017-10-03 21:50:40,885 INFO [ProcExecWrkr-16] hbase.MetaTableAccessor(1770): Updated table testHBase3583 state to DISABLING in META 2017-10-03 21:50:40,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 48948ms 2017-10-03 21:50:40,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 177171ms 2017-10-03 21:50:41,071 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testHBase3583, region=4d1729514f896c7adccdd0a78f0eb9bc, server=cn012.l42scl.hortonworks.com,37573,1507067090360}] 2017-10-03 21:50:41,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=38 2017-10-03 21:50:41,246 INFO [ProcExecWrkr-16] procedure.MasterProcedureScheduler(814): pid=39, ppid=38, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testHBase3583, region=4d1729514f896c7adccdd0a78f0eb9bc, server=cn012.l42scl.hortonworks.com,37573,1507067090360 testHBase3583 testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. 2017-10-03 21:50:41,247 INFO [ProcExecWrkr-16] assignment.RegionStateStore(186): pid=39 updating hbase:meta row=testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc., regionState=CLOSING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:41,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:41,253 INFO [ProcExecWrkr-16] assignment.RegionTransitionProcedure(213): Dispatch pid=39, ppid=38, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testHBase3583, region=4d1729514f896c7adccdd0a78f0eb9bc, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:41,403 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:50:41,406 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1520): Close 4d1729514f896c7adccdd0a78f0eb9bc without moving 2017-10-03 21:50:41,407 DEBUG [RS_CLOSE_REGION-cn012:37573-2] handler.CloseRegionHandler(91): Processing close of testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. 2017-10-03 21:50:41,407 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(1560): Closing testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc.: disabling compactions & flushes 2017-10-03 21:50:41,408 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(1599): Updates disabled for region testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. 2017-10-03 21:50:41,408 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(2535): Flushing 3/3 column families, memstore=30 B 2017-10-03 21:50:41,408 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc.store: a 2017-10-03 21:50:41,409 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:50:41,409 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc.store: b 2017-10-03 21:50:41,409 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:0, and the number of cells in new segment is:0 2017-10-03 21:50:41,409 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc.store: c 2017-10-03 21:50:41,409 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:0, and the number of cells in new segment is:0 2017-10-03 21:50:41,430 INFO [IPC Server handler 8 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741855_1031{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 0 2017-10-03 21:50:41,432 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=5, memsize=30, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/.tmp/a/01588d388240456c9ba6859c9eabea4b 2017-10-03 21:50:41,442 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/.tmp/a/01588d388240456c9ba6859c9eabea4b as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/a/01588d388240456c9ba6859c9eabea4b 2017-10-03 21:50:41,450 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/a/01588d388240456c9ba6859c9eabea4b, entries=1, sequenceid=5, filesize=4.6 K 2017-10-03 21:50:41,452 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(2731): Finished memstore flush of ~30 B/30, currentsize=0 B/0 for region testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. in 44ms, sequenceid=5, compaction requested=false 2017-10-03 21:50:41,453 INFO [StoreCloserThread-testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc.-1] regionserver.HStore(926): Closed a 2017-10-03 21:50:41,453 INFO [StoreCloserThread-testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc.-1] regionserver.HStore(926): Closed b 2017-10-03 21:50:41,454 INFO [StoreCloserThread-testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc.-1] regionserver.HStore(926): Closed c 2017-10-03 21:50:41,459 DEBUG [RS_CLOSE_REGION-cn012:37573-2] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/recovered.edits/8.seqid, newSeqId=8, maxSeqId=2 2017-10-03 21:50:41,460 DEBUG [RS_CLOSE_REGION-cn012:37573-2] coprocessor.CoprocessorHost(291): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2017-10-03 21:50:41,460 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(1712): Closed testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. 2017-10-03 21:50:41,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report CLOSED seqId=-1, pid=39, ppid=38, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testHBase3583, region=4d1729514f896c7adccdd0a78f0eb9bc, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:41,463 DEBUG [RS_CLOSE_REGION-cn012:37573-2] handler.CloseRegionHandler(123): Closed testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. 2017-10-03 21:50:41,463 INFO [ProcExecWrkr-20] assignment.RegionStateStore(186): pid=39 updating hbase:meta row=testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc., regionState=CLOSED 2017-10-03 21:50:41,597 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(1978): Worker stuck ProcExecWrkr-9(pid=35) run time 4mins, 59.529sec 2017-10-03 21:50:41,621 INFO [ProcExecWrkr-20] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=39, ppid=38, state=SUCCESS; UnassignProcedure table=testHBase3583, region=4d1729514f896c7adccdd0a78f0eb9bc, server=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:41,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=38 2017-10-03 21:50:41,771 INFO [ProcExecWrkr-20] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=38, state=RUNNABLE:DISABLE_TABLE_SET_DISABLED_TABLE_STATE; DisableTableProcedure table=testHBase3583; resume parent processing. 2017-10-03 21:50:41,771 INFO [ProcExecWrkr-20] procedure2.ProcedureExecutor(1246): Finished pid=39, ppid=38, state=SUCCESS; UnassignProcedure table=testHBase3583, region=4d1729514f896c7adccdd0a78f0eb9bc, server=cn012.l42scl.hortonworks.com,37573,1507067090360 in 396msec 2017-10-03 21:50:41,771 DEBUG [ProcExecWrkr-18] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"testHBase3583","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067441771}]}} 2017-10-03 21:50:41,776 INFO [ProcExecWrkr-18] hbase.MetaTableAccessor(1770): Updated table testHBase3583 state to DISABLED in META 2017-10-03 21:50:41,777 INFO [ProcExecWrkr-18] procedure.DisableTableProcedure(285): Disabled table, testHBase3583, is completed. 2017-10-03 21:50:41,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 298025ms 2017-10-03 21:50:41,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 287451ms 2017-10-03 21:50:42,062 INFO [ProcExecWrkr-18] procedure2.ProcedureExecutor(1246): Finished pid=38, state=SUCCESS; DisableTableProcedure table=testHBase3583 in 1.5160sec 2017-10-03 21:50:42,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=0, pid=33, ppid=32, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:42,309 INFO [ProcExecWrkr-9] procedure2.ProcedureExecutor(1340): Rolled back pid=35, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.exceptions.TimeoutIOException via master-delete-table:org.apache.hadoop.hbase.exceptions.TimeoutIOException: Timed out while waiting on regions TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. in transition; DeleteTableProcedure table=TestTable.testRecovery exec-time=5mins, 0.253sec 2017-10-03 21:50:42,309 INFO [ProcExecWrkr-9] assignment.RegionStateStore(186): pid=33 updating hbase:meta row=TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a., regionState=OPEN, openSeqNum=0, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:42,409 INFO [ProcExecWrkr-9] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=33, ppid=32, state=SUCCESS; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a 2017-10-03 21:50:42,462 INFO [ProcExecWrkr-9] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=32, state=RUNNABLE:SERVER_CRASH_FINISH; ServerCrashProcedure server=cn012.l42scl.hortonworks.com,43156,1507067138439, splitWal=true, meta=false; resume parent processing. 2017-10-03 21:50:42,462 INFO [ProcExecWrkr-9] procedure2.ProcedureExecutor(1246): Finished pid=33, ppid=32, state=SUCCESS; AssignProcedure table=TestTable.testRecovery, region=e8d466bc47137a0d2d7b3119106dc88a in 5mins, 1.042sec 2017-10-03 21:50:42,531 INFO [ProcExecWrkr-19] procedure2.ProcedureExecutor(1246): Finished pid=32, state=SUCCESS; ServerCrashProcedure server=cn012.l42scl.hortonworks.com,43156,1507067138439, splitWal=true, meta=false in 5mins, 1.611sec 2017-10-03 21:50:42,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=38 2017-10-03 21:50:42,643 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DISABLE, Table Name: default:testHBase3583 completed 2017-10-03 21:50:42,644 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$5(2109): Client=hbase//172.18.128.12 delete testHBase3583 2017-10-03 21:50:42,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=40, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testHBase3583 2017-10-03 21:50:42,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=40 2017-10-03 21:50:42,792 DEBUG [ProcExecWrkr-21] procedure.DeleteTableProcedure(99): Waiting for 'testHBase3583' regions in transition 2017-10-03 21:50:42,837 DEBUG [ProcExecWrkr-21] procedure.DeleteTableProcedure(110): delete 'testHBase3583' regions from META 2017-10-03 21:50:42,838 DEBUG [ProcExecWrkr-21] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067442837}]},"ts":9223372036854775807} 2017-10-03 21:50:42,843 INFO [ProcExecWrkr-21] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:50:42,843 DEBUG [ProcExecWrkr-21] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => 4d1729514f896c7adccdd0a78f0eb9bc, NAME => 'testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:50:42,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=40 2017-10-03 21:50:42,911 DEBUG [ProcExecWrkr-21] procedure.DeleteTableProcedure(115): delete 'testHBase3583' from filesystem 2017-10-03 21:50:42,914 DEBUG [ProcExecWrkr-21] procedure.DeleteTableProcedure(315): Archiving region testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc. from FS 2017-10-03 21:50:42,914 DEBUG [ProcExecWrkr-21] backup.HFileArchiver(114): ARCHIVING hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc 2017-10-03 21:50:42,918 DEBUG [ProcExecWrkr-21] backup.HFileArchiver(155): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/recovered.edits] 2017-10-03 21:50:42,924 DEBUG [ProcExecWrkr-21] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/a/01588d388240456c9ba6859c9eabea4b, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/a/01588d388240456c9ba6859c9eabea4b 2017-10-03 21:50:42,931 DEBUG [ProcExecWrkr-21] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/recovered.edits/8.seqid, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/recovered.edits/8.seqid 2017-10-03 21:50:42,931 INFO [IPC Server handler 6 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741854_1030 127.0.0.1:41361 2017-10-03 21:50:42,933 DEBUG [ProcExecWrkr-21] backup.HFileArchiver(466): Deleted hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc 2017-10-03 21:50:42,933 DEBUG [ProcExecWrkr-21] procedure.DeleteTableProcedure(319): Table 'testHBase3583' archived! 2017-10-03 21:50:42,934 INFO [IPC Server handler 3 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741853_1029 127.0.0.1:41361 2017-10-03 21:50:43,000 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 270658ms 2017-10-03 21:50:43,000 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 172903ms 2017-10-03 21:50:43,011 DEBUG [ProcExecWrkr-21] procedure.DeleteTableProcedure(121): delete 'testHBase3583' descriptor 2017-10-03 21:50:43,011 DEBUG [ProcExecWrkr-21] procedure.DeleteTableProcedure(399): Removing 'testHBase3583' descriptor. 2017-10-03 21:50:43,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=40 2017-10-03 21:50:43,112 DEBUG [ProcExecWrkr-21] procedure.DeleteTableProcedure(126): delete 'testHBase3583' assignment state 2017-10-03 21:50:43,113 DEBUG [ProcExecWrkr-21] procedure.DeleteTableProcedure(389): Removing 'testHBase3583' from region states. 2017-10-03 21:50:43,113 DEBUG [ProcExecWrkr-21] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067443113}]},"ts":9223372036854775807} 2017-10-03 21:50:43,117 INFO [ProcExecWrkr-21] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:50:43,117 DEBUG [ProcExecWrkr-21] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => 4d1729514f896c7adccdd0a78f0eb9bc, NAME => 'testHBase3583,,1507067436136.4d1729514f896c7adccdd0a78f0eb9bc.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:50:43,117 DEBUG [ProcExecWrkr-21] procedure.DeleteTableProcedure(393): Marking 'testHBase3583' as deleted. 2017-10-03 21:50:43,118 DEBUG [ProcExecWrkr-21] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"testHBase3583","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":1507067443118}]},"ts":9223372036854775807} 2017-10-03 21:50:43,120 INFO [ProcExecWrkr-21] hbase.MetaTableAccessor(1796): Deleted table testHBase3583 state from META 2017-10-03 21:50:43,162 DEBUG [ProcExecWrkr-21] procedure.DeleteTableProcedure(132): delete 'testHBase3583' completed 2017-10-03 21:50:43,195 INFO [ProcExecWrkr-21] procedure2.ProcedureExecutor(1246): Finished pid=40, state=SUCCESS; DeleteTableProcedure table=testHBase3583 in 517msec 2017-10-03 21:50:43,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=40 2017-10-03 21:50:43,396 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DELETE, Table Name: default:testHBase3583 completed 2017-10-03 21:50:43,402 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@70efb718] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:41361 to delete [blk_1073741853_1029, blk_1073741854_1030] 2017-10-03 21:50:43,427 INFO [main] hbase.ResourceChecker(172): after: coprocessor.TestRegionObserverInterface#testHBase3583 Thread=273 (was 253) - Thread LEAK? -, OpenFileDescriptor=1244 (was 1226) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32000 (was 32000), SystemLoadAverage=421 (was 432), ProcessCount=349 (was 347) - ProcessCount LEAK? -, AvailableMemoryMB=49538 (was 48351) - AvailableMemoryMB LEAK? - 2017-10-03 21:50:43,428 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1244 is superior to 1024 2017-10-03 21:50:43,457 INFO [main] hbase.ResourceChecker(148): before: coprocessor.TestRegionObserverInterface#testHBase3758 Thread=272, OpenFileDescriptor=1244, MaxFileDescriptor=32000, SystemLoadAverage=421, ProcessCount=349, AvailableMemoryMB=49536 2017-10-03 21:50:43,457 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1244 is superior to 1024 2017-10-03 21:50:43,462 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$3(1748): Client=hbase//172.18.128.12 create 'testHBase3758', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2017-10-03 21:50:43,506 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2017-10-03 21:50:43,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=41, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testHBase3758 2017-10-03 21:50:43,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=41 2017-10-03 21:50:43,685 INFO [IPC Server handler 8 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741856_1032{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|FINALIZED]]} size 0 2017-10-03 21:50:43,688 DEBUG [ProcExecWrkr-22] util.FSTableDescriptors(728): Wrote descriptor into: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3758/.tabledesc/.tableinfo.0000000001 2017-10-03 21:50:43,690 INFO [RegionOpenAndInitThread-testHBase3758-1] regionserver.HRegion(6477): creating HRegion testHBase3758 HTD == 'testHBase3758', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp Table name == testHBase3758 2017-10-03 21:50:43,702 INFO [IPC Server handler 2 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741857_1033{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|FINALIZED]]} size 0 2017-10-03 21:50:43,703 DEBUG [RegionOpenAndInitThread-testHBase3758-1] regionserver.HRegion(798): Instantiated testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. 2017-10-03 21:50:43,704 DEBUG [RegionOpenAndInitThread-testHBase3758-1] regionserver.HRegion(1560): Closing testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3.: disabling compactions & flushes 2017-10-03 21:50:43,704 DEBUG [RegionOpenAndInitThread-testHBase3758-1] regionserver.HRegion(1599): Updates disabled for region testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. 2017-10-03 21:50:43,704 INFO [RegionOpenAndInitThread-testHBase3758-1] regionserver.HRegion(1712): Closed testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. 2017-10-03 21:50:43,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=41 2017-10-03 21:50:43,757 DEBUG [ProcExecWrkr-22] hbase.MetaTableAccessor(1417): Put{"totalColumns":1,"row":"testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3.","families":{"info":[{"qualifier":"regioninfo","vlen":47,"tag":[],"timestamp":9223372036854775807}]}} 2017-10-03 21:50:43,761 INFO [ProcExecWrkr-22] hbase.MetaTableAccessor(1614): Added 1 2017-10-03 21:50:43,804 DEBUG [ProcExecWrkr-22] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"testHBase3758","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067443804}]}} 2017-10-03 21:50:43,808 INFO [ProcExecWrkr-22] hbase.MetaTableAccessor(1770): Updated table testHBase3758 state to ENABLING in META 2017-10-03 21:50:43,809 INFO [ProcExecWrkr-22] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=42, ppid=41, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testHBase3758, region=3404fa33032497dabbb84bdbaa63afc3}] 2017-10-03 21:50:43,874 INFO [ProcExecWrkr-22] procedure.MasterProcedureScheduler(814): pid=42, ppid=41, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testHBase3758, region=3404fa33032497dabbb84bdbaa63afc3 testHBase3758 testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. 2017-10-03 21:50:43,877 INFO [ProcExecWrkr-22] assignment.AssignProcedure(204): Start pid=42, ppid=41, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testHBase3758, region=3404fa33032497dabbb84bdbaa63afc3; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2017-10-03 21:50:43,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=41 2017-10-03 21:50:43,952 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 259152ms 2017-10-03 21:50:43,952 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 168552ms 2017-10-03 21:50:44,029 INFO [ProcExecWrkr-1] assignment.RegionStateStore(186): pid=42 updating hbase:meta row=testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3., regionState=OPENING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:44,038 INFO [ProcExecWrkr-1] assignment.RegionTransitionProcedure(213): Dispatch pid=42, ppid=41, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testHBase3758, region=3404fa33032497dabbb84bdbaa63afc3; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:44,188 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:50:44,190 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1916): Open testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. 2017-10-03 21:50:44,196 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(6654): Opening region: {ENCODED => 3404fa33032497dabbb84bdbaa63afc3, NAME => 'testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3.', STARTKEY => '', ENDKEY => ''} 2017-10-03 21:50:44,197 INFO [RS_OPEN_REGION-cn012:37573-1] coprocessor.CoprocessorHost(164): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2017-10-03 21:50:44,197 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testHBase3758 3404fa33032497dabbb84bdbaa63afc3 2017-10-03 21:50:44,197 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(798): Instantiated testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. 2017-10-03 21:50:44,203 DEBUG [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/a 2017-10-03 21:50:44,203 DEBUG [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/a 2017-10-03 21:50:44,204 INFO [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:50:44,204 INFO [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:50:44,205 INFO [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] hfile.CacheConfig(258): Created cacheConfig for a: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:50:44,205 INFO [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:50:44,209 DEBUG [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/b 2017-10-03 21:50:44,209 DEBUG [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/b 2017-10-03 21:50:44,209 INFO [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:50:44,210 INFO [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:50:44,210 INFO [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] hfile.CacheConfig(258): Created cacheConfig for b: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:50:44,210 INFO [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:50:44,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=41 2017-10-03 21:50:44,215 DEBUG [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/c 2017-10-03 21:50:44,216 DEBUG [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/c 2017-10-03 21:50:44,216 INFO [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:50:44,217 INFO [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:50:44,217 INFO [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] hfile.CacheConfig(258): Created cacheConfig for c: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:50:44,217 INFO [StoreOpener-3404fa33032497dabbb84bdbaa63afc3-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:50:44,219 INFO [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(1030): Setting FlushNonSloppyStoresFirstPolicy for the region=testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. 2017-10-03 21:50:44,220 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(4121): Found 0 recovered edits file(s) under hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3 2017-10-03 21:50:44,223 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table testHBase3758, use config (44739242) instead 2017-10-03 21:50:44,227 DEBUG [RS_OPEN_REGION-cn012:37573-1] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2017-10-03 21:50:44,227 INFO [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(957): Onlined 3404fa33032497dabbb84bdbaa63afc3; next sequenceid=2 2017-10-03 21:50:44,229 INFO [PostOpenDeployTasks:3404fa33032497dabbb84bdbaa63afc3] regionserver.HRegionServer(2170): Post open deploy tasks for testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. 2017-10-03 21:50:44,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=42, ppid=41, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testHBase3758, region=3404fa33032497dabbb84bdbaa63afc3; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:44,231 DEBUG [PostOpenDeployTasks:3404fa33032497dabbb84bdbaa63afc3] regionserver.HRegionServer(2197): Finished post open deploy task for testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. 2017-10-03 21:50:44,232 INFO [ProcExecWrkr-24] assignment.RegionStateStore(186): pid=42 updating hbase:meta row=testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3., regionState=OPEN, openSeqNum=2, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:44,234 DEBUG [RS_OPEN_REGION-cn012:37573-1] handler.OpenRegionHandler(127): Opened testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. on cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:44,321 INFO [ProcExecWrkr-24] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=42, ppid=41, state=SUCCESS; AssignProcedure table=testHBase3758, region=3404fa33032497dabbb84bdbaa63afc3 2017-10-03 21:50:44,439 INFO [ProcExecWrkr-24] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=41, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testHBase3758; resume parent processing. 2017-10-03 21:50:44,440 INFO [ProcExecWrkr-24] procedure2.ProcedureExecutor(1246): Finished pid=42, ppid=41, state=SUCCESS; AssignProcedure table=testHBase3758, region=3404fa33032497dabbb84bdbaa63afc3 in 430msec 2017-10-03 21:50:44,440 DEBUG [ProcExecWrkr-2] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"testHBase3758","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067444439}]}} 2017-10-03 21:50:44,444 INFO [ProcExecWrkr-2] hbase.MetaTableAccessor(1770): Updated table testHBase3758 state to ENABLED in META 2017-10-03 21:50:44,604 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1246): Finished pid=41, state=SUCCESS; CreateTableProcedure table=testHBase3758 in 1.0500sec 2017-10-03 21:50:44,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=41 2017-10-03 21:50:44,715 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: CREATE, Table Name: default:testHBase3758 completed 2017-10-03 21:50:44,715 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:50:44,721 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:50:44,733 INFO [Time-limited test] client.HBaseAdmin$15(919): Started disable of testHBase3758 2017-10-03 21:50:44,734 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$11(2331): Client=hbase//172.18.128.12 disable testHBase3758 2017-10-03 21:50:44,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testHBase3758 2017-10-03 21:50:44,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=43 2017-10-03 21:50:44,982 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 114524ms 2017-10-03 21:50:44,982 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 188177ms 2017-10-03 21:50:45,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=43 2017-10-03 21:50:45,058 DEBUG [ProcExecWrkr-4] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"testHBase3758","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067445058}]}} 2017-10-03 21:50:45,064 INFO [ProcExecWrkr-4] hbase.MetaTableAccessor(1770): Updated table testHBase3758 state to DISABLING in META 2017-10-03 21:50:45,129 INFO [ProcExecWrkr-4] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testHBase3758, region=3404fa33032497dabbb84bdbaa63afc3, server=cn012.l42scl.hortonworks.com,37573,1507067090360}] 2017-10-03 21:50:45,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=43 2017-10-03 21:50:45,212 INFO [ProcExecWrkr-4] procedure.MasterProcedureScheduler(814): pid=44, ppid=43, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testHBase3758, region=3404fa33032497dabbb84bdbaa63afc3, server=cn012.l42scl.hortonworks.com,37573,1507067090360 testHBase3758 testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. 2017-10-03 21:50:45,213 INFO [ProcExecWrkr-4] assignment.RegionStateStore(186): pid=44 updating hbase:meta row=testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3., regionState=CLOSING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:45,217 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(213): Dispatch pid=44, ppid=43, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testHBase3758, region=3404fa33032497dabbb84bdbaa63afc3, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:45,367 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:50:45,369 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1520): Close 3404fa33032497dabbb84bdbaa63afc3 without moving 2017-10-03 21:50:45,369 DEBUG [RS_CLOSE_REGION-cn012:37573-0] handler.CloseRegionHandler(91): Processing close of testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. 2017-10-03 21:50:45,371 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(1560): Closing testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3.: disabling compactions & flushes 2017-10-03 21:50:45,371 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(1599): Updates disabled for region testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. 2017-10-03 21:50:45,371 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(2535): Flushing 3/3 column families, memstore=114 B 2017-10-03 21:50:45,372 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3.store: a 2017-10-03 21:50:45,372 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:50:45,372 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3.store: b 2017-10-03 21:50:45,372 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:50:45,372 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3.store: c 2017-10-03 21:50:45,373 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:50:45,392 INFO [IPC Server handler 0 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741858_1034{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 4802 2017-10-03 21:50:45,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=43 2017-10-03 21:50:45,793 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=6, memsize=58, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/.tmp/a/2b41178f315343de899ae77c413225d7 2017-10-03 21:50:45,802 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.StoreFileReader(497): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2b41178f315343de899ae77c413225d7 2017-10-03 21:50:45,815 INFO [IPC Server handler 8 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741859_1035{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 4802 2017-10-03 21:50:45,991 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 200668ms 2017-10-03 21:50:45,991 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 283873ms 2017-10-03 21:50:46,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=43 2017-10-03 21:50:46,217 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=6, memsize=28, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/.tmp/b/ab9a4b9043cc44098e54b07ee927fb0b 2017-10-03 21:50:46,226 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.StoreFileReader(497): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ab9a4b9043cc44098e54b07ee927fb0b 2017-10-03 21:50:46,241 INFO [IPC Server handler 4 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741860_1036{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 4802 2017-10-03 21:50:46,384 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2017-10-03 21:50:46,643 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=6, memsize=28, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/.tmp/c/e37f899a0118485186ea318e8e602844 2017-10-03 21:50:46,651 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.StoreFileReader(497): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e37f899a0118485186ea318e8e602844 2017-10-03 21:50:46,653 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/.tmp/a/2b41178f315343de899ae77c413225d7 as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/a/2b41178f315343de899ae77c413225d7 2017-10-03 21:50:46,660 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.StoreFileReader(497): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2b41178f315343de899ae77c413225d7 2017-10-03 21:50:46,661 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/a/2b41178f315343de899ae77c413225d7, entries=1, sequenceid=6, filesize=4.7 K 2017-10-03 21:50:46,662 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/.tmp/b/ab9a4b9043cc44098e54b07ee927fb0b as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/b/ab9a4b9043cc44098e54b07ee927fb0b 2017-10-03 21:50:46,669 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.StoreFileReader(497): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ab9a4b9043cc44098e54b07ee927fb0b 2017-10-03 21:50:46,669 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/b/ab9a4b9043cc44098e54b07ee927fb0b, entries=1, sequenceid=6, filesize=4.7 K 2017-10-03 21:50:46,671 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/.tmp/c/e37f899a0118485186ea318e8e602844 as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/c/e37f899a0118485186ea318e8e602844 2017-10-03 21:50:46,677 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.StoreFileReader(497): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e37f899a0118485186ea318e8e602844 2017-10-03 21:50:46,678 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/c/e37f899a0118485186ea318e8e602844, entries=1, sequenceid=6, filesize=4.7 K 2017-10-03 21:50:46,679 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(2731): Finished memstore flush of ~114 B/114, currentsize=0 B/0 for region testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. in 1308ms, sequenceid=6, compaction requested=false 2017-10-03 21:50:46,681 INFO [StoreCloserThread-testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3.-1] regionserver.HStore(926): Closed a 2017-10-03 21:50:46,681 INFO [StoreCloserThread-testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3.-1] regionserver.HStore(926): Closed b 2017-10-03 21:50:46,682 INFO [StoreCloserThread-testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3.-1] regionserver.HStore(926): Closed c 2017-10-03 21:50:46,688 DEBUG [RS_CLOSE_REGION-cn012:37573-0] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/recovered.edits/9.seqid, newSeqId=9, maxSeqId=2 2017-10-03 21:50:46,688 DEBUG [RS_CLOSE_REGION-cn012:37573-0] coprocessor.CoprocessorHost(291): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2017-10-03 21:50:46,689 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(1712): Closed testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. 2017-10-03 21:50:46,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report CLOSED seqId=-1, pid=44, ppid=43, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testHBase3758, region=3404fa33032497dabbb84bdbaa63afc3, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:46,691 DEBUG [RS_CLOSE_REGION-cn012:37573-0] handler.CloseRegionHandler(123): Closed testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. 2017-10-03 21:50:46,691 INFO [ProcExecWrkr-5] assignment.RegionStateStore(186): pid=44 updating hbase:meta row=testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3., regionState=CLOSED 2017-10-03 21:50:46,761 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=44, ppid=43, state=SUCCESS; UnassignProcedure table=testHBase3758, region=3404fa33032497dabbb84bdbaa63afc3, server=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:46,812 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=43, state=RUNNABLE:DISABLE_TABLE_SET_DISABLED_TABLE_STATE; DisableTableProcedure table=testHBase3758; resume parent processing. 2017-10-03 21:50:46,813 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1246): Finished pid=44, ppid=43, state=SUCCESS; UnassignProcedure table=testHBase3758, region=3404fa33032497dabbb84bdbaa63afc3, server=cn012.l42scl.hortonworks.com,37573,1507067090360 in 1.5670sec 2017-10-03 21:50:46,813 DEBUG [ProcExecWrkr-3] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"testHBase3758","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067446812}]}} 2017-10-03 21:50:46,817 INFO [ProcExecWrkr-3] hbase.MetaTableAccessor(1770): Updated table testHBase3758 state to DISABLED in META 2017-10-03 21:50:46,817 INFO [ProcExecWrkr-3] procedure.DisableTableProcedure(285): Disabled table, testHBase3758, is completed. 2017-10-03 21:50:46,904 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1246): Finished pid=43, state=SUCCESS; DisableTableProcedure table=testHBase3758 in 2.1280sec 2017-10-03 21:50:46,921 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 153388ms 2017-10-03 21:50:46,921 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 292628ms 2017-10-03 21:50:47,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=43 2017-10-03 21:50:47,010 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DISABLE, Table Name: default:testHBase3758 completed 2017-10-03 21:50:47,011 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$5(2109): Client=hbase//172.18.128.12 delete testHBase3758 2017-10-03 21:50:47,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=45, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testHBase3758 2017-10-03 21:50:47,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=45 2017-10-03 21:50:47,225 DEBUG [ProcExecWrkr-7] procedure.DeleteTableProcedure(99): Waiting for 'testHBase3758' regions in transition 2017-10-03 21:50:47,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=45 2017-10-03 21:50:47,360 DEBUG [ProcExecWrkr-7] procedure.DeleteTableProcedure(110): delete 'testHBase3758' regions from META 2017-10-03 21:50:47,360 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067447360}]},"ts":9223372036854775807} 2017-10-03 21:50:47,364 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:50:47,364 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => 3404fa33032497dabbb84bdbaa63afc3, NAME => 'testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:50:47,429 DEBUG [ProcExecWrkr-7] procedure.DeleteTableProcedure(115): delete 'testHBase3758' from filesystem 2017-10-03 21:50:47,432 DEBUG [ProcExecWrkr-7] procedure.DeleteTableProcedure(315): Archiving region testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3. from FS 2017-10-03 21:50:47,432 DEBUG [ProcExecWrkr-7] backup.HFileArchiver(114): ARCHIVING hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3 2017-10-03 21:50:47,435 DEBUG [ProcExecWrkr-7] backup.HFileArchiver(155): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/recovered.edits] 2017-10-03 21:50:47,439 DEBUG [ProcExecWrkr-7] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/a/2b41178f315343de899ae77c413225d7, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/a/2b41178f315343de899ae77c413225d7 2017-10-03 21:50:47,443 DEBUG [ProcExecWrkr-7] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/b/ab9a4b9043cc44098e54b07ee927fb0b, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/b/ab9a4b9043cc44098e54b07ee927fb0b 2017-10-03 21:50:47,447 DEBUG [ProcExecWrkr-7] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/c/e37f899a0118485186ea318e8e602844, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/c/e37f899a0118485186ea318e8e602844 2017-10-03 21:50:47,450 DEBUG [ProcExecWrkr-7] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/recovered.edits/9.seqid, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/recovered.edits/9.seqid 2017-10-03 21:50:47,451 INFO [IPC Server handler 0 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741857_1033 127.0.0.1:41361 2017-10-03 21:50:47,452 DEBUG [ProcExecWrkr-7] backup.HFileArchiver(466): Deleted hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3 2017-10-03 21:50:47,452 DEBUG [ProcExecWrkr-7] procedure.DeleteTableProcedure(319): Table 'testHBase3758' archived! 2017-10-03 21:50:47,454 INFO [IPC Server handler 2 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741856_1032 127.0.0.1:41361 2017-10-03 21:50:47,504 DEBUG [ProcExecWrkr-7] procedure.DeleteTableProcedure(121): delete 'testHBase3758' descriptor 2017-10-03 21:50:47,504 DEBUG [ProcExecWrkr-7] procedure.DeleteTableProcedure(399): Removing 'testHBase3758' descriptor. 2017-10-03 21:50:47,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=45 2017-10-03 21:50:47,571 DEBUG [ProcExecWrkr-7] procedure.DeleteTableProcedure(126): delete 'testHBase3758' assignment state 2017-10-03 21:50:47,571 DEBUG [ProcExecWrkr-7] procedure.DeleteTableProcedure(389): Removing 'testHBase3758' from region states. 2017-10-03 21:50:47,571 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067447571}]},"ts":9223372036854775807} 2017-10-03 21:50:47,575 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:50:47,576 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => 3404fa33032497dabbb84bdbaa63afc3, NAME => 'testHBase3758,,1507067443462.3404fa33032497dabbb84bdbaa63afc3.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:50:47,576 DEBUG [ProcExecWrkr-7] procedure.DeleteTableProcedure(393): Marking 'testHBase3758' as deleted. 2017-10-03 21:50:47,576 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"testHBase3758","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":1507067447576}]},"ts":9223372036854775807} 2017-10-03 21:50:47,579 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1796): Deleted table testHBase3758 state from META 2017-10-03 21:50:47,638 DEBUG [ProcExecWrkr-7] procedure.DeleteTableProcedure(132): delete 'testHBase3758' completed 2017-10-03 21:50:47,671 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1246): Finished pid=45, state=SUCCESS; DeleteTableProcedure table=testHBase3758 in 626msec 2017-10-03 21:50:47,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=45 2017-10-03 21:50:47,827 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DELETE, Table Name: default:testHBase3758 completed 2017-10-03 21:50:47,852 INFO [main] hbase.ResourceChecker(172): after: coprocessor.TestRegionObserverInterface#testHBase3758 Thread=281 (was 272) - Thread LEAK? -, OpenFileDescriptor=1244 (was 1244), MaxFileDescriptor=32000 (was 32000), SystemLoadAverage=396 (was 421), ProcessCount=347 (was 349), AvailableMemoryMB=48867 (was 49536) 2017-10-03 21:50:47,852 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1244 is superior to 1024 2017-10-03 21:50:47,879 INFO [main] hbase.ResourceChecker(148): before: coprocessor.TestRegionObserverInterface#testCompactionOverride Thread=281, OpenFileDescriptor=1244, MaxFileDescriptor=32000, SystemLoadAverage=396, ProcessCount=347, AvailableMemoryMB=48865 2017-10-03 21:50:47,879 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1244 is superior to 1024 2017-10-03 21:50:47,886 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$3(1748): Client=hbase//172.18.128.12 create 'testCompactionOverride', {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor|1073741823|'}, {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2017-10-03 21:50:47,990 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 225265ms 2017-10-03 21:50:47,990 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 288423ms 2017-10-03 21:50:48,304 WARN [DataXceiver for client DFSClient_NONMAPREDUCE_-174551704_1 at /127.0.0.1:58812 [Receiving block BP-608549805-172.18.128.12-1507067084534:blk_1073741829_1005]] datanode.BlockReceiver(391): Slow flushOrSync took 316ms (threshold=300ms), isSync:true, flushTotalNanos=15330ns 2017-10-03 21:50:48,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=46, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCompactionOverride 2017-10-03 21:50:48,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=46 2017-10-03 21:50:48,371 INFO [IPC Server handler 0 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741861_1037{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 629 2017-10-03 21:50:48,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=46 2017-10-03 21:50:48,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=46 2017-10-03 21:50:48,776 DEBUG [ProcExecWrkr-8] util.FSTableDescriptors(728): Wrote descriptor into: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testCompactionOverride/.tabledesc/.tableinfo.0000000001 2017-10-03 21:50:48,780 INFO [RegionOpenAndInitThread-testCompactionOverride-1] regionserver.HRegion(6477): creating HRegion testCompactionOverride HTD == 'testCompactionOverride', {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor|1073741823|'}, {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp Table name == testCompactionOverride 2017-10-03 21:50:48,804 INFO [IPC Server handler 3 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741862_1038{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 57 2017-10-03 21:50:48,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 165810ms 2017-10-03 21:50:48,919 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 176466ms 2017-10-03 21:50:48,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=46 2017-10-03 21:50:49,207 DEBUG [RegionOpenAndInitThread-testCompactionOverride-1] regionserver.HRegion(798): Instantiated testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. 2017-10-03 21:50:49,207 DEBUG [RegionOpenAndInitThread-testCompactionOverride-1] regionserver.HRegion(1560): Closing testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781.: disabling compactions & flushes 2017-10-03 21:50:49,207 DEBUG [RegionOpenAndInitThread-testCompactionOverride-1] regionserver.HRegion(1599): Updates disabled for region testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. 2017-10-03 21:50:49,207 INFO [RegionOpenAndInitThread-testCompactionOverride-1] regionserver.HRegion(1712): Closed testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. 2017-10-03 21:50:49,288 DEBUG [ProcExecWrkr-8] hbase.MetaTableAccessor(1417): Put{"totalColumns":1,"row":"testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":9223372036854775807}]}} 2017-10-03 21:50:49,294 INFO [ProcExecWrkr-8] hbase.MetaTableAccessor(1614): Added 1 2017-10-03 21:50:49,369 DEBUG [ProcExecWrkr-8] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"testCompactionOverride","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067449369}]}} 2017-10-03 21:50:49,375 INFO [ProcExecWrkr-8] hbase.MetaTableAccessor(1770): Updated table testCompactionOverride state to ENABLING in META 2017-10-03 21:50:49,375 INFO [ProcExecWrkr-8] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=47, ppid=46, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testCompactionOverride, region=338d120f0bd69660aa21ecb994fb9781}] 2017-10-03 21:50:49,402 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@70efb718] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:41361 to delete [blk_1073741856_1032, blk_1073741857_1033] 2017-10-03 21:50:49,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=46 2017-10-03 21:50:49,438 INFO [ProcExecWrkr-8] procedure.MasterProcedureScheduler(814): pid=47, ppid=46, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testCompactionOverride, region=338d120f0bd69660aa21ecb994fb9781 testCompactionOverride testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. 2017-10-03 21:50:49,441 INFO [ProcExecWrkr-8] assignment.AssignProcedure(204): Start pid=47, ppid=46, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testCompactionOverride, region=338d120f0bd69660aa21ecb994fb9781; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2017-10-03 21:50:49,594 INFO [ProcExecWrkr-11] assignment.RegionStateStore(186): pid=47 updating hbase:meta row=testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781., regionState=OPENING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:49,605 INFO [ProcExecWrkr-11] assignment.RegionTransitionProcedure(213): Dispatch pid=47, ppid=46, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testCompactionOverride, region=338d120f0bd69660aa21ecb994fb9781; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:49,701 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2017-10-03 21:50:49,755 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:50:49,757 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1916): Open testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. 2017-10-03 21:50:49,762 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(6654): Opening region: {ENCODED => 338d120f0bd69660aa21ecb994fb9781, NAME => 'testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781.', STARTKEY => '', ENDKEY => ''} 2017-10-03 21:50:49,763 INFO [RS_OPEN_REGION-cn012:37573-2] coprocessor.CoprocessorHost(164): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2017-10-03 21:50:49,763 DEBUG [RS_OPEN_REGION-cn012:37573-2] coprocessor.CoprocessorHost(205): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor with path null and priority 1073741823 2017-10-03 21:50:49,763 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.RegionCoprocessorHost(379): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor from HTD of testCompactionOverride successfully. 2017-10-03 21:50:49,763 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testCompactionOverride 338d120f0bd69660aa21ecb994fb9781 2017-10-03 21:50:49,763 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(798): Instantiated testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. 2017-10-03 21:50:49,768 DEBUG [StoreOpener-338d120f0bd69660aa21ecb994fb9781-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781/a 2017-10-03 21:50:49,768 DEBUG [StoreOpener-338d120f0bd69660aa21ecb994fb9781-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781/a 2017-10-03 21:50:49,769 INFO [StoreOpener-338d120f0bd69660aa21ecb994fb9781-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 33554432 2017-10-03 21:50:49,769 INFO [StoreOpener-338d120f0bd69660aa21ecb994fb9781-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:50:49,769 INFO [StoreOpener-338d120f0bd69660aa21ecb994fb9781-1] hfile.CacheConfig(258): Created cacheConfig for a: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:50:49,769 INFO [StoreOpener-338d120f0bd69660aa21ecb994fb9781-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:50:49,770 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(1030): Setting FlushNonSloppyStoresFirstPolicy for the region=testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. 2017-10-03 21:50:49,772 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(4121): Found 0 recovered edits file(s) under hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781 2017-10-03 21:50:49,773 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table testCompactionOverride, use config (134217728) instead 2017-10-03 21:50:49,776 DEBUG [RS_OPEN_REGION-cn012:37573-2] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2017-10-03 21:50:49,776 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(957): Onlined 338d120f0bd69660aa21ecb994fb9781; next sequenceid=2 2017-10-03 21:50:49,778 INFO [PostOpenDeployTasks:338d120f0bd69660aa21ecb994fb9781] regionserver.HRegionServer(2170): Post open deploy tasks for testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. 2017-10-03 21:50:49,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=47, ppid=46, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testCompactionOverride, region=338d120f0bd69660aa21ecb994fb9781; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:49,780 INFO [ProcExecWrkr-12] assignment.RegionStateStore(186): pid=47 updating hbase:meta row=testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781., regionState=OPEN, openSeqNum=2, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:49,781 DEBUG [PostOpenDeployTasks:338d120f0bd69660aa21ecb994fb9781] regionserver.HRegionServer(2197): Finished post open deploy task for testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. 2017-10-03 21:50:49,781 DEBUG [RS_OPEN_REGION-cn012:37573-2] handler.OpenRegionHandler(127): Opened testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. on cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:49,821 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=47, ppid=46, state=SUCCESS; AssignProcedure table=testCompactionOverride, region=338d120f0bd69660aa21ecb994fb9781 2017-10-03 21:50:49,862 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=46, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testCompactionOverride; resume parent processing. 2017-10-03 21:50:49,862 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1246): Finished pid=47, ppid=46, state=SUCCESS; AssignProcedure table=testCompactionOverride, region=338d120f0bd69660aa21ecb994fb9781 in 409msec 2017-10-03 21:50:49,863 DEBUG [ProcExecWrkr-15] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"testCompactionOverride","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067449862}]}} 2017-10-03 21:50:49,866 INFO [ProcExecWrkr-15] hbase.MetaTableAccessor(1770): Updated table testCompactionOverride state to ENABLED in META 2017-10-03 21:50:49,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 113784ms 2017-10-03 21:50:49,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 4729ms 2017-10-03 21:50:49,982 INFO [ProcExecWrkr-15] procedure2.ProcedureExecutor(1246): Finished pid=46, state=SUCCESS; CreateTableProcedure table=testCompactionOverride in 2.0260sec 2017-10-03 21:50:50,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=46 2017-10-03 21:50:50,425 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: CREATE, Table Name: default:testCompactionOverride completed 2017-10-03 21:50:50,432 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.HRegion(7859): writing data to region testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. with WAL disabled. Data may be lost in the event of a crash. 2017-10-03 21:50:50,462 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(821): Client=hbase//172.18.128.12 procedure request for: flush-table-proc 2017-10-03 21:50:50,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure.ProcedureCoordinator(177): Submitting procedure testCompactionOverride 2017-10-03 21:50:50,472 INFO [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'testCompactionOverride' 2017-10-03 21:50:50,472 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2017-10-03 21:50:50,473 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'testCompactionOverride' starting 'acquire' 2017-10-03 21:50:50,473 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'testCompactionOverride', kicking off acquire phase on members. 2017-10-03 21:50:50,474 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(364): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testCompactionOverride 2017-10-03 21:50:50,474 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testCompactionOverride 2017-10-03 21:50:50,506 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2017-10-03 21:50:50,506 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testCompactionOverride/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,506 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2017-10-03 21:50:50,507 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(150): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2017-10-03 21:50:50,507 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(364): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testCompactionOverride/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,507 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2017-10-03 21:50:50,507 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(182): Found procedure znode: /hbase/flush-table-proc/acquired/testCompactionOverride 2017-10-03 21:50:50,508 DEBUG [main-EventThread] zookeeper.ZKUtil(364): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testCompactionOverride 2017-10-03 21:50:50,509 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(208): start proc data length is 4 2017-10-03 21:50:50,509 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): Found data for znode:/hbase/flush-table-proc/acquired/testCompactionOverride 2017-10-03 21:50:50,510 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testCompactionOverride 2017-10-03 21:50:50,513 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:testCompactionOverride 2017-10-03 21:50:50,514 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] procedure.Subprocedure(157): Starting subprocedure 'testCompactionOverride' with timeout 60000ms 2017-10-03 21:50:50,514 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2017-10-03 21:50:50,514 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] procedure.Subprocedure(165): Subprocedure 'testCompactionOverride' starting 'acquire' stage 2017-10-03 21:50:50,515 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2017-10-03 21:50:50,515 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(244): Waiting for local region flush to finish. 2017-10-03 21:50:50,515 DEBUG [rs(cn012.l42scl.hortonworks.com,37573,1507067090360)-flush-proc-pool9-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. 2017-10-03 21:50:50,516 DEBUG [rs(cn012.l42scl.hortonworks.com,37573,1507067090360)-flush-proc-pool9-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. started... 2017-10-03 21:50:50,516 INFO [rs(cn012.l42scl.hortonworks.com,37573,1507067090360)-flush-proc-pool9-thread-1] regionserver.HRegion(2535): Flushing 1/1 column families, memstore=380 B 2017-10-03 21:50:50,517 DEBUG [rs(cn012.l42scl.hortonworks.com,37573,1507067090360)-flush-proc-pool9-thread-1] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781.store: a 2017-10-03 21:50:50,517 DEBUG [rs(cn012.l42scl.hortonworks.com,37573,1507067090360)-flush-proc-pool9-thread-1] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:50:50,532 INFO [IPC Server handler 9 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741863_1039{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 0 2017-10-03 21:50:50,533 INFO [rs(cn012.l42scl.hortonworks.com,37573,1507067090360)-flush-proc-pool9-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=14, memsize=380, hasBloomFilter=true, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781/.tmp/a/166bdda8b1124a1d86c3ca69d46fc310 2017-10-03 21:50:50,542 DEBUG [rs(cn012.l42scl.hortonworks.com,37573,1507067090360)-flush-proc-pool9-thread-1] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781/.tmp/a/166bdda8b1124a1d86c3ca69d46fc310 as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781/a/166bdda8b1124a1d86c3ca69d46fc310 2017-10-03 21:50:50,552 INFO [rs(cn012.l42scl.hortonworks.com,37573,1507067090360)-flush-proc-pool9-thread-1] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781/a/166bdda8b1124a1d86c3ca69d46fc310, entries=10, sequenceid=14, filesize=5.1 K 2017-10-03 21:50:50,554 INFO [rs(cn012.l42scl.hortonworks.com,37573,1507067090360)-flush-proc-pool9-thread-1] regionserver.HRegion(2731): Finished memstore flush of ~380 B/380, currentsize=0 B/0 for region testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. in 38ms, sequenceid=14, compaction requested=false 2017-10-03 21:50:50,554 DEBUG [rs(cn012.l42scl.hortonworks.com,37573,1507067090360)-flush-proc-pool9-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. 2017-10-03 21:50:50,554 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1/1 local region flush tasks. 2017-10-03 21:50:50,555 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(257): Completed 1 local region flush tasks. 2017-10-03 21:50:50,555 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(290): cancelling 0 flush region tasks cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,555 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] procedure.Subprocedure(167): Subprocedure 'testCompactionOverride' locally acquired 2017-10-03 21:50:50,556 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(239): Member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' joining acquired barrier for procedure (testCompactionOverride) in zk 2017-10-03 21:50:50,564 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testCompactionOverride/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,564 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(247): Watch for global barrier reached:/hbase/flush-table-proc/reached/testCompactionOverride 2017-10-03 21:50:50,564 DEBUG [main-EventThread] procedure.ZKProcedureCoordinator$1(195): Node created: /hbase/flush-table-proc/acquired/testCompactionOverride/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,565 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2017-10-03 21:50:50,565 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] zookeeper.ZKUtil(364): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testCompactionOverride 2017-10-03 21:50:50,565 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2017-10-03 21:50:50,565 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] procedure.Subprocedure(172): Subprocedure 'testCompactionOverride' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2017-10-03 21:50:50,565 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2017-10-03 21:50:50,565 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2017-10-03 21:50:50,566 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testCompactionOverride 2017-10-03 21:50:50,566 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,566 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2017-10-03 21:50:50,567 DEBUG [main-EventThread] procedure.Procedure(298): member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' joining acquired barrier for procedure 'testCompactionOverride' on coordinator 2017-10-03 21:50:50,567 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'testCompactionOverride' starting 'in-barrier' execution. 2017-10-03 21:50:50,567 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@689a1d3e[Count = 0] remaining members to acquire global barrier 2017-10-03 21:50:50,567 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testCompactionOverride 2017-10-03 21:50:50,573 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testCompactionOverride 2017-10-03 21:50:50,573 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testCompactionOverride 2017-10-03 21:50:50,573 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(364): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testCompactionOverride/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,573 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testCompactionOverride 2017-10-03 21:50:50,573 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2017-10-03 21:50:50,574 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] procedure.Subprocedure(186): Subprocedure 'testCompactionOverride' received 'reached' from coordinator. 2017-10-03 21:50:50,574 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] procedure.Subprocedure(188): Subprocedure 'testCompactionOverride' locally completed 2017-10-03 21:50:50,574 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(263): Marking procedure 'testCompactionOverride' completed for member 'cn012.l42scl.hortonworks.com,37573,1507067090360' in zk 2017-10-03 21:50:50,602 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testCompactionOverride/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,602 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] procedure.Subprocedure(193): Subprocedure 'testCompactionOverride' has notified controller of completion 2017-10-03 21:50:50,602 DEBUG [main-EventThread] procedure.ZKProcedureCoordinator$1(195): Node created: /hbase/flush-table-proc/reached/testCompactionOverride/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,602 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2017-10-03 21:50:50,602 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2017-10-03 21:50:50,602 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2017-10-03 21:50:50,602 DEBUG [member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' subprocedure-pool2-thread-1] procedure.Subprocedure(218): Subprocedure 'testCompactionOverride' completed. 2017-10-03 21:50:50,603 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2017-10-03 21:50:50,603 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2017-10-03 21:50:50,603 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testCompactionOverride 2017-10-03 21:50:50,604 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,604 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2017-10-03 21:50:50,604 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testCompactionOverride 2017-10-03 21:50:50,605 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,605 DEBUG [main-EventThread] procedure.ZKProcedureCoordinator$1(220): Finished data from procedure 'testCompactionOverride' member 'cn012.l42scl.hortonworks.com,37573,1507067090360': 2017-10-03 21:50:50,605 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'cn012.l42scl.hortonworks.com,37573,1507067090360' released barrier for procedure'testCompactionOverride', counting down latch. Waiting for 0 more 2017-10-03 21:50:50,605 INFO [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'testCompactionOverride' execution completed 2017-10-03 21:50:50,606 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2017-10-03 21:50:50,606 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2017-10-03 21:50:50,606 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinator(163): Attempting to clean out zk node for op:testCompactionOverride 2017-10-03 21:50:50,606 INFO [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure testCompactionOverrideincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2017-10-03 21:50:50,633 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testCompactionOverride 2017-10-03 21:50:50,633 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testCompactionOverride 2017-10-03 21:50:50,633 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testCompactionOverride 2017-10-03 21:50:50,633 DEBUG [main-EventThread] procedure.ZKProcedureCoordinator$1(195): Node created: /hbase/flush-table-proc/abort/testCompactionOverride 2017-10-03 21:50:50,633 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(311): Aborting procedure member for znode /hbase/flush-table-proc/abort/testCompactionOverride 2017-10-03 21:50:50,633 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2017-10-03 21:50:50,634 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2017-10-03 21:50:50,634 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2017-10-03 21:50:50,634 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2017-10-03 21:50:50,634 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2017-10-03 21:50:50,634 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2017-10-03 21:50:50,634 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(362): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testCompactionOverride/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,635 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(311): Aborting procedure member for znode /hbase/flush-table-proc/abort/testCompactionOverride 2017-10-03 21:50:50,635 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testCompactionOverride 2017-10-03 21:50:50,635 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2017-10-03 21:50:50,636 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testCompactionOverride 2017-10-03 21:50:50,636 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,636 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(362): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testCompactionOverride/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,637 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2017-10-03 21:50:50,637 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testCompactionOverride 2017-10-03 21:50:50,637 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,656 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testCompactionOverride/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,656 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testCompactionOverride 2017-10-03 21:50:50,656 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2017-10-03 21:50:50,656 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testCompactionOverride 2017-10-03 21:50:50,656 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2017-10-03 21:50:50,656 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testCompactionOverride/cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:50,656 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(150): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2017-10-03 21:50:50,656 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testCompactionOverride 2017-10-03 21:50:50,657 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testCompactionOverride 2017-10-03 21:50:50,657 DEBUG [(cn012.l42scl.hortonworks.com,35273,1507067089134)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2017-10-03 21:50:50,657 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testCompactionOverride 2017-10-03 21:50:50,658 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'testCompactionOverride' 2017-10-03 21:50:50,658 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2017-10-03 21:50:50,658 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2017-10-03 21:50:50,658 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2017-10-03 21:50:50,658 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2017-10-03 21:50:50,661 DEBUG [Time-limited test] client.HBaseAdmin(2829): Waiting a max of 300000 ms for procedure 'flush-table-proc : testCompactionOverride'' to complete. (max 8571 ms per retry) 2017-10-03 21:50:50,661 DEBUG [Time-limited test] client.HBaseAdmin(2838): (#1) Sleeping: 100ms while waiting for procedure completion. 2017-10-03 21:50:50,761 DEBUG [Time-limited test] client.HBaseAdmin(2844): Getting current status of procedure from master... 2017-10-03 21:50:50,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1081): Checking to see if procedure from request:flush-table-proc is done 2017-10-03 21:50:50,773 DEBUG [Time-limited test] coprocessor.TestRegionObserverInterface(520): Flush complete 2017-10-03 21:50:50,780 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(122): Process identifier=hbase-admin-on-hconnection-0x61a91c9b connecting to ZooKeeper ensemble=localhost:57257 2017-10-03 21:50:50,789 DEBUG [Time-limited test] ipc.NettyRpcConnection(255): Connecting to cn012.l42scl.hortonworks.com/172.18.128.12:37573 2017-10-03 21:50:50,794 INFO [Reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=37573] ipc.ServerRpcConnection(528): Auth successful for hbase (auth:SIMPLE) 2017-10-03 21:50:50,795 INFO [Reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=37573] ipc.ServerRpcConnection(558): Connection from 172.18.128.12 port: 39402 with version info: version: "2.0.0-alpha3" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/2-hbase" revision: "d3a817f212f5d012f3cc7c6277314b682a907b0b" user: "hbase" date: "Tue Oct 3 21:04:59 UTC 2017" src_checksum: "0a7d6891d3586c77eaed645726da640b" version_major: 2 version_minor: 0 2017-10-03 21:50:50,795 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1557): Compacting testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. 2017-10-03 21:50:50,797 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=37573] compactions.SortedCompactionPolicy(69): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 10 blocking 2017-10-03 21:50:50,798 DEBUG [Time-limited test-EventThread] zookeeper.ZooKeeperWatcher(465): hbase-admin-on-hconnection-0x61a91c9b0x0, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2017-10-03 21:50:50,799 DEBUG [Time-limited test-EventThread] zookeeper.ZooKeeperWatcher(528): hbase-admin-on-hconnection-0x61a91c9b-0x15ee432cd9b000e connected 2017-10-03 21:50:50,800 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.HStore(1741): 338d120f0bd69660aa21ecb994fb9781 - a: Initiating major compaction (all files) 2017-10-03 21:50:50,802 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.CompactSplit(299): Small Compaction requested: Optional[org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@32feb46f]; Because: User-triggered major compaction; compaction_queue=(0:0), split_queue=0 2017-10-03 21:50:50,802 INFO [RS:0;cn012:37573-shortCompactions-1507067450802] regionserver.HRegion(2128): Starting compaction on a in region testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. 2017-10-03 21:50:50,804 INFO [RS:0;cn012:37573-shortCompactions-1507067450802] regionserver.HStore(1368): Starting compaction of [hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781/a/166bdda8b1124a1d86c3ca69d46fc310] into tmpdir=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781/.tmp, totalSize=5.1 K 2017-10-03 21:50:50,806 DEBUG [RS:0;cn012:37573-shortCompactions-1507067450802] compactions.Compactor(203): Compacting hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781/a/166bdda8b1124a1d86c3ca69d46fc310, keycount=10, bloomtype=ROW, size=5.1 K, encoding=NONE, seqNum=14, earliestPutTs=1507067450432 2017-10-03 21:50:50,830 INFO [RS:0;cn012:37573-shortCompactions-1507067450802] throttle.PressureAwareThroughputController(152): testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781.#a#compaction#11 average throughput is 0.18 MB/sec, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 10.00 MB/sec 2017-10-03 21:50:50,838 INFO [IPC Server handler 2 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741864_1040{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 5033 2017-10-03 21:50:50,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 38191ms 2017-10-03 21:50:50,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 177494ms 2017-10-03 21:50:51,249 DEBUG [RS:0;cn012:37573-shortCompactions-1507067450802] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781/.tmp/a/3f22223fa49141d69a928f9aef939152 as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781/a/3f22223fa49141d69a928f9aef939152 2017-10-03 21:50:51,266 INFO [RS:0;cn012:37573-shortCompactions-1507067450802] regionserver.HStore(1507): Completed major compaction of 1 (all) file(s) in a of 338d120f0bd69660aa21ecb994fb9781 into 3f22223fa49141d69a928f9aef939152(size=4.9 K), total size for store is 4.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2017-10-03 21:50:51,271 INFO [RS:0;cn012:37573-shortCompactions-1507067450802] regionserver.CompactSplit$CompactionRunner(531): Completed compaction: Request = regionName=testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781., storeName=a, fileCount=1, fileSize=5.1 K (5.1 K), priority=1, time=1507067450799; duration=0sec 2017-10-03 21:50:51,274 DEBUG [RS:0;cn012:37573-shortCompactions-1507067450802] regionserver.CompactSplit$CompactionRunner(558): CompactSplitThread Status: compaction_queue=(0:0), split_queue=0 2017-10-03 21:50:51,771 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(220): Evict completed pid=32, state=SUCCESS; ServerCrashProcedure server=cn012.l42scl.hortonworks.com,43156,1507067138439, splitWal=true, meta=false 2017-10-03 21:50:51,773 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(220): Evict completed pid=34, state=SUCCESS; DisableTableProcedure table=TestTable.testRecovery 2017-10-03 21:50:51,773 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(220): Evict completed pid=19, state=SUCCESS; DisableTableProcedure table=SKIPPED_BY_PREWALRESTORE 2017-10-03 21:50:51,773 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(220): Evict completed pid=21, state=SUCCESS; DeleteTableProcedure table=SKIPPED_BY_PREWALRESTORE 2017-10-03 21:50:51,773 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(220): Evict completed pid=22, state=SUCCESS; CreateTableProcedure table=TestTable.bulkLoadHFileTest 2017-10-03 21:50:51,773 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(220): Evict completed pid=24, state=SUCCESS; DisableTableProcedure table=TestTable.bulkLoadHFileTest 2017-10-03 21:50:51,774 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(220): Evict completed pid=26, state=SUCCESS; DeleteTableProcedure table=TestTable.bulkLoadHFileTest 2017-10-03 21:50:51,774 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(220): Evict completed pid=27, state=SUCCESS; CreateTableProcedure table=TestTable.testRecovery 2017-10-03 21:50:51,814 DEBUG [Time-limited test] coprocessor.TestRegionObserverInterface(531): Last compaction was at 1507067451256 2017-10-03 21:50:51,877 INFO [main] hbase.ResourceChecker(172): after: coprocessor.TestRegionObserverInterface#testCompactionOverride Thread=289 (was 281) - Thread LEAK? -, OpenFileDescriptor=1246 (was 1244) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32000 (was 32000), SystemLoadAverage=388 (was 396), ProcessCount=347 (was 347), AvailableMemoryMB=48718 (was 48865) 2017-10-03 21:50:51,877 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1246 is superior to 1024 2017-10-03 21:50:51,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 185564ms 2017-10-03 21:50:51,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 61129ms 2017-10-03 21:50:51,923 INFO [main] hbase.ResourceChecker(148): before: coprocessor.TestRegionObserverInterface#testCheckAndPutHooks Thread=289, OpenFileDescriptor=1246, MaxFileDescriptor=32000, SystemLoadAverage=388, ProcessCount=347, AvailableMemoryMB=48716 2017-10-03 21:50:51,923 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1246 is superior to 1024 2017-10-03 21:50:51,929 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$3(1748): Client=hbase//172.18.128.12 create 'TestTable.testCheckAndPutHooks', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2017-10-03 21:50:52,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=48, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestTable.testCheckAndPutHooks 2017-10-03 21:50:52,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=48 2017-10-03 21:50:52,167 INFO [IPC Server handler 8 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741865_1041{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 0 2017-10-03 21:50:52,172 DEBUG [ProcExecWrkr-13] util.FSTableDescriptors(728): Wrote descriptor into: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testCheckAndPutHooks/.tabledesc/.tableinfo.0000000001 2017-10-03 21:50:52,175 INFO [RegionOpenAndInitThread-TestTable.testCheckAndPutHooks-1] regionserver.HRegion(6477): creating HRegion TestTable.testCheckAndPutHooks HTD == 'TestTable.testCheckAndPutHooks', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp Table name == TestTable.testCheckAndPutHooks 2017-10-03 21:50:52,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=48 2017-10-03 21:50:52,190 INFO [IPC Server handler 2 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741866_1042{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|RBW]]} size 65 2017-10-03 21:50:52,385 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2017-10-03 21:50:52,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=48 2017-10-03 21:50:52,592 DEBUG [RegionOpenAndInitThread-TestTable.testCheckAndPutHooks-1] regionserver.HRegion(798): Instantiated TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. 2017-10-03 21:50:52,593 DEBUG [RegionOpenAndInitThread-TestTable.testCheckAndPutHooks-1] regionserver.HRegion(1560): Closing TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506.: disabling compactions & flushes 2017-10-03 21:50:52,593 DEBUG [RegionOpenAndInitThread-TestTable.testCheckAndPutHooks-1] regionserver.HRegion(1599): Updates disabled for region TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. 2017-10-03 21:50:52,593 INFO [RegionOpenAndInitThread-TestTable.testCheckAndPutHooks-1] regionserver.HRegion(1712): Closed TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. 2017-10-03 21:50:52,665 DEBUG [ProcExecWrkr-13] hbase.MetaTableAccessor(1417): Put{"totalColumns":1,"row":"TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":9223372036854775807}]}} 2017-10-03 21:50:52,670 INFO [ProcExecWrkr-13] hbase.MetaTableAccessor(1614): Added 1 2017-10-03 21:50:52,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=48 2017-10-03 21:50:52,713 DEBUG [ProcExecWrkr-13] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testCheckAndPutHooks","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067452712}]}} 2017-10-03 21:50:52,717 INFO [ProcExecWrkr-13] hbase.MetaTableAccessor(1770): Updated table TestTable.testCheckAndPutHooks state to ENABLING in META 2017-10-03 21:50:52,717 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testCheckAndPutHooks, region=eac18e9cf3db3c83e2236ac664f87506}] 2017-10-03 21:50:52,786 INFO [ProcExecWrkr-13] procedure.MasterProcedureScheduler(814): pid=49, ppid=48, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testCheckAndPutHooks, region=eac18e9cf3db3c83e2236ac664f87506 TestTable.testCheckAndPutHooks TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. 2017-10-03 21:50:52,789 INFO [ProcExecWrkr-13] assignment.AssignProcedure(204): Start pid=49, ppid=48, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testCheckAndPutHooks, region=eac18e9cf3db3c83e2236ac664f87506; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2017-10-03 21:50:52,941 INFO [ProcExecWrkr-17] assignment.RegionStateStore(186): pid=49 updating hbase:meta row=TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506., regionState=OPENING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:52,949 INFO [ProcExecWrkr-17] assignment.RegionTransitionProcedure(213): Dispatch pid=49, ppid=48, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testCheckAndPutHooks, region=eac18e9cf3db3c83e2236ac664f87506; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:52,951 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 18864ms 2017-10-03 21:50:52,951 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 273773ms 2017-10-03 21:50:53,099 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:50:53,101 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1916): Open TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. 2017-10-03 21:50:53,109 DEBUG [RS_OPEN_REGION-cn012:37573-0] regionserver.HRegion(6654): Opening region: {ENCODED => eac18e9cf3db3c83e2236ac664f87506, NAME => 'TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506.', STARTKEY => '', ENDKEY => ''} 2017-10-03 21:50:53,109 INFO [RS_OPEN_REGION-cn012:37573-0] coprocessor.CoprocessorHost(164): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2017-10-03 21:50:53,110 DEBUG [RS_OPEN_REGION-cn012:37573-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table TestTable.testCheckAndPutHooks eac18e9cf3db3c83e2236ac664f87506 2017-10-03 21:50:53,110 DEBUG [RS_OPEN_REGION-cn012:37573-0] regionserver.HRegion(798): Instantiated TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. 2017-10-03 21:50:53,116 DEBUG [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/a 2017-10-03 21:50:53,116 DEBUG [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/a 2017-10-03 21:50:53,116 INFO [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:50:53,116 INFO [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:50:53,116 INFO [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] hfile.CacheConfig(258): Created cacheConfig for a: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:50:53,117 INFO [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:50:53,121 DEBUG [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/b 2017-10-03 21:50:53,121 DEBUG [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/b 2017-10-03 21:50:53,122 INFO [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:50:53,122 INFO [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:50:53,122 INFO [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] hfile.CacheConfig(258): Created cacheConfig for b: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:50:53,123 INFO [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:50:53,126 DEBUG [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/c 2017-10-03 21:50:53,127 DEBUG [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/c 2017-10-03 21:50:53,127 INFO [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:50:53,127 INFO [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:50:53,127 INFO [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] hfile.CacheConfig(258): Created cacheConfig for c: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:50:53,127 INFO [StoreOpener-eac18e9cf3db3c83e2236ac664f87506-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:50:53,129 INFO [RS_OPEN_REGION-cn012:37573-0] regionserver.HRegion(1030): Setting FlushNonSloppyStoresFirstPolicy for the region=TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. 2017-10-03 21:50:53,130 DEBUG [RS_OPEN_REGION-cn012:37573-0] regionserver.HRegion(4121): Found 0 recovered edits file(s) under hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506 2017-10-03 21:50:53,133 DEBUG [RS_OPEN_REGION-cn012:37573-0] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table TestTable.testCheckAndPutHooks, use config (44739242) instead 2017-10-03 21:50:53,138 DEBUG [RS_OPEN_REGION-cn012:37573-0] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2017-10-03 21:50:53,138 INFO [RS_OPEN_REGION-cn012:37573-0] regionserver.HRegion(957): Onlined eac18e9cf3db3c83e2236ac664f87506; next sequenceid=2 2017-10-03 21:50:53,142 INFO [PostOpenDeployTasks:eac18e9cf3db3c83e2236ac664f87506] regionserver.HRegionServer(2170): Post open deploy tasks for TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. 2017-10-03 21:50:53,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=49, ppid=48, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testCheckAndPutHooks, region=eac18e9cf3db3c83e2236ac664f87506; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:53,144 DEBUG [PostOpenDeployTasks:eac18e9cf3db3c83e2236ac664f87506] regionserver.HRegionServer(2197): Finished post open deploy task for TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. 2017-10-03 21:50:53,144 INFO [ProcExecWrkr-16] assignment.RegionStateStore(186): pid=49 updating hbase:meta row=TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506., regionState=OPEN, openSeqNum=2, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:53,145 DEBUG [RS_OPEN_REGION-cn012:37573-0] handler.OpenRegionHandler(127): Opened TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. on cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:53,187 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=49, ppid=48, state=SUCCESS; AssignProcedure table=TestTable.testCheckAndPutHooks, region=eac18e9cf3db3c83e2236ac664f87506 2017-10-03 21:50:53,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=48 2017-10-03 21:50:53,259 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=48, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=TestTable.testCheckAndPutHooks; resume parent processing. 2017-10-03 21:50:53,259 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1246): Finished pid=49, ppid=48, state=SUCCESS; AssignProcedure table=TestTable.testCheckAndPutHooks, region=eac18e9cf3db3c83e2236ac664f87506 in 432msec 2017-10-03 21:50:53,259 DEBUG [ProcExecWrkr-20] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testCheckAndPutHooks","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067453259}]}} 2017-10-03 21:50:53,264 INFO [ProcExecWrkr-20] hbase.MetaTableAccessor(1770): Updated table TestTable.testCheckAndPutHooks state to ENABLED in META 2017-10-03 21:50:53,538 INFO [ProcExecWrkr-20] procedure2.ProcedureExecutor(1246): Finished pid=48, state=SUCCESS; CreateTableProcedure table=TestTable.testCheckAndPutHooks in 1.4430sec 2017-10-03 21:50:53,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 200708ms 2017-10-03 21:50:53,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 163308ms 2017-10-03 21:50:54,148 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data; isDirectory=true; modification_time=1507067103338; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:50:54,149 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default; isDirectory=true; modification_time=1507067447435; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:50:54,156 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3758; isDirectory=true; modification_time=1507067447435; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks; isDirectory=true; modification_time=1507067103338; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest; isDirectory=true; modification_time=1507067135109; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3583; isDirectory=true; modification_time=1507067442919; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE; isDirectory=true; modification_time=1507067129674; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:50:54,157 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3; isDirectory=true; modification_time=1507067447449; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:50:54,162 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/a; isDirectory=true; modification_time=1507067447439; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/b; isDirectory=true; modification_time=1507067447443; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/c; isDirectory=true; modification_time=1507067447446; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3758/3404fa33032497dabbb84bdbaa63afc3/recovered.edits; isDirectory=true; modification_time=1507067447450; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:50:54,175 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e; isDirectory=true; modification_time=1507067103366; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:50:54,179 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/a; isDirectory=true; modification_time=1507067103348; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/b; isDirectory=true; modification_time=1507067103356; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/c; isDirectory=true; modification_time=1507067103362; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/recovered.edits; isDirectory=true; modification_time=1507067103369; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:50:54,183 DEBUG [cn012:35273.masterManager-HFileCleaner.small.0-1507067091581] cleaner.HFileCleaner(248): Removing: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/a/441701d735834b44bfe0db94fb446a5d from archive 2017-10-03 21:50:54,185 INFO [IPC Server handler 3 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741836_1012 127.0.0.1:41361 2017-10-03 21:50:54,190 DEBUG [cn012:35273.masterManager-HFileCleaner.small.0-1507067091581] cleaner.HFileCleaner(248): Removing: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/b/dbd28f076aee4cf19e58bc3fa683d64b from archive 2017-10-03 21:50:54,191 INFO [IPC Server handler 6 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741837_1013 127.0.0.1:41361 2017-10-03 21:50:54,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=48 2017-10-03 21:50:54,194 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: CREATE, Table Name: default:TestTable.testCheckAndPutHooks completed 2017-10-03 21:50:54,195 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:50:54,195 DEBUG [cn012:35273.masterManager-HFileCleaner.small.0-1507067091581] cleaner.HFileCleaner(248): Removing: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/c/f0299bedab0e4f45ab410c2649a5f03f from archive 2017-10-03 21:50:54,196 INFO [IPC Server handler 4 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741838_1014 127.0.0.1:41361 2017-10-03 21:50:54,200 DEBUG [cn012:35273.masterManager-HFileCleaner.small.0-1507067091581] cleaner.HFileCleaner(248): Removing: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndDeleteHooks/a2c67b38fa1700ae384334c352b27d2e/recovered.edits/9.seqid from archive 2017-10-03 21:50:54,202 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:50:54,203 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44; isDirectory=true; modification_time=1507067135125; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:50:54,205 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44/a; isDirectory=true; modification_time=1507067135117; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44/recovered.edits; isDirectory=true; modification_time=1507067135129; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:50:54,207 DEBUG [cn012:35273.masterManager-HFileCleaner.small.0-1507067091581] cleaner.HFileCleaner(248): Removing: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44/a/6327b38babd74f858527c08898229609_SeqId_4_ from archive 2017-10-03 21:50:54,210 INFO [IPC Server handler 3 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741845_1021 127.0.0.1:41361 2017-10-03 21:50:54,215 DEBUG [cn012:35273.masterManager-HFileCleaner.small.0-1507067091581] cleaner.HFileCleaner(248): Removing: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.bulkLoadHFileTest/7438af971b0419af15efde0dafafdb44/recovered.edits/6.seqid from archive 2017-10-03 21:50:54,218 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc; isDirectory=true; modification_time=1507067442929; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:50:54,219 INFO [Time-limited test] client.HBaseAdmin$15(919): Started disable of TestTable.testCheckAndPutHooks 2017-10-03 21:50:54,220 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$11(2331): Client=hbase//172.18.128.12 disable TestTable.testCheckAndPutHooks 2017-10-03 21:50:54,220 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/a; isDirectory=true; modification_time=1507067442924; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}, FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBase3583/4d1729514f896c7adccdd0a78f0eb9bc/recovered.edits; isDirectory=true; modification_time=1507067442930; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:50:54,225 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE/65972395a311743485849c968cac9876; isDirectory=true; modification_time=1507067129682; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:50:54,226 DEBUG [cn012.l42scl.hortonworks.com,35273,1507067089134_Chore_1] cleaner.CleanerChore(223): Prepared to delete files in directories: [FileStatus{path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE/65972395a311743485849c968cac9876/recovered.edits; isDirectory=true; modification_time=1507067129686; access_time=0; owner=hbase; group=supergroup; permission=rwxr-xr-x; isSymlink=false}] 2017-10-03 21:50:54,228 DEBUG [cn012:35273.masterManager-HFileCleaner.small.0-1507067091581] cleaner.HFileCleaner(248): Removing: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/SKIPPED_BY_PREWALRESTORE/65972395a311743485849c968cac9876/recovered.edits/10.seqid from archive 2017-10-03 21:50:54,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestTable.testCheckAndPutHooks 2017-10-03 21:50:54,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=50 2017-10-03 21:50:54,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=50 2017-10-03 21:50:54,543 DEBUG [ProcExecWrkr-18] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testCheckAndPutHooks","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067454542}]}} 2017-10-03 21:50:54,547 INFO [ProcExecWrkr-18] hbase.MetaTableAccessor(1770): Updated table TestTable.testCheckAndPutHooks state to DISABLING in META 2017-10-03 21:50:54,670 INFO [ProcExecWrkr-18] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testCheckAndPutHooks, region=eac18e9cf3db3c83e2236ac664f87506, server=cn012.l42scl.hortonworks.com,37573,1507067090360}] 2017-10-03 21:50:54,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=50 2017-10-03 21:50:54,732 INFO [ProcExecWrkr-9] procedure.MasterProcedureScheduler(814): pid=51, ppid=50, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testCheckAndPutHooks, region=eac18e9cf3db3c83e2236ac664f87506, server=cn012.l42scl.hortonworks.com,37573,1507067090360 TestTable.testCheckAndPutHooks TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. 2017-10-03 21:50:54,733 INFO [ProcExecWrkr-9] assignment.RegionStateStore(186): pid=51 updating hbase:meta row=TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506., regionState=CLOSING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:54,738 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(213): Dispatch pid=51, ppid=50, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testCheckAndPutHooks, region=eac18e9cf3db3c83e2236ac664f87506, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:54,888 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:50:54,891 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1520): Close eac18e9cf3db3c83e2236ac664f87506 without moving 2017-10-03 21:50:54,891 DEBUG [RS_CLOSE_REGION-cn012:37573-1] handler.CloseRegionHandler(91): Processing close of TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. 2017-10-03 21:50:54,893 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(1560): Closing TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506.: disabling compactions & flushes 2017-10-03 21:50:54,893 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(1599): Updates disabled for region TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. 2017-10-03 21:50:54,893 INFO [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(2535): Flushing 3/3 column families, memstore=54 B 2017-10-03 21:50:54,893 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506.store: a 2017-10-03 21:50:54,894 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:50:54,894 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506.store: b 2017-10-03 21:50:54,894 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:0, and the number of cells in new segment is:0 2017-10-03 21:50:54,894 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506.store: c 2017-10-03 21:50:54,895 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:0, and the number of cells in new segment is:0 2017-10-03 21:50:54,916 INFO [IPC Server handler 2 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741867_1043{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 4660 2017-10-03 21:50:54,922 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 143381ms 2017-10-03 21:50:54,922 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 54254ms 2017-10-03 21:50:55,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=50 2017-10-03 21:50:55,266 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2017-10-03 21:50:55,318 INFO [RS_CLOSE_REGION-cn012:37573-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=6, memsize=54, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/.tmp/a/f99d6d663d8e4fd9bae8c78a9cb3d684 2017-10-03 21:50:55,331 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/.tmp/a/f99d6d663d8e4fd9bae8c78a9cb3d684 as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/a/f99d6d663d8e4fd9bae8c78a9cb3d684 2017-10-03 21:50:55,341 INFO [RS_CLOSE_REGION-cn012:37573-1] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/a/f99d6d663d8e4fd9bae8c78a9cb3d684, entries=1, sequenceid=6, filesize=4.6 K 2017-10-03 21:50:55,343 INFO [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(2731): Finished memstore flush of ~54 B/54, currentsize=0 B/0 for region TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. in 450ms, sequenceid=6, compaction requested=false 2017-10-03 21:50:55,345 INFO [StoreCloserThread-TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506.-1] regionserver.HStore(926): Closed a 2017-10-03 21:50:55,346 INFO [StoreCloserThread-TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506.-1] regionserver.HStore(926): Closed b 2017-10-03 21:50:55,346 INFO [StoreCloserThread-TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506.-1] regionserver.HStore(926): Closed c 2017-10-03 21:50:55,353 DEBUG [RS_CLOSE_REGION-cn012:37573-1] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/recovered.edits/9.seqid, newSeqId=9, maxSeqId=2 2017-10-03 21:50:55,355 DEBUG [RS_CLOSE_REGION-cn012:37573-1] coprocessor.CoprocessorHost(291): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2017-10-03 21:50:55,355 INFO [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(1712): Closed TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. 2017-10-03 21:50:55,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report CLOSED seqId=-1, pid=51, ppid=50, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testCheckAndPutHooks, region=eac18e9cf3db3c83e2236ac664f87506, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:55,357 DEBUG [RS_CLOSE_REGION-cn012:37573-1] handler.CloseRegionHandler(123): Closed TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. 2017-10-03 21:50:55,357 INFO [ProcExecWrkr-19] assignment.RegionStateStore(186): pid=51 updating hbase:meta row=TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506., regionState=CLOSED 2017-10-03 21:50:55,403 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@70efb718] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:41361 to delete [blk_1073741845_1021, blk_1073741836_1012, blk_1073741837_1013, blk_1073741838_1014] 2017-10-03 21:50:55,454 INFO [ProcExecWrkr-19] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=51, ppid=50, state=SUCCESS; UnassignProcedure table=TestTable.testCheckAndPutHooks, region=eac18e9cf3db3c83e2236ac664f87506, server=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:55,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=50 2017-10-03 21:50:55,601 INFO [ProcExecWrkr-19] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=50, state=RUNNABLE:DISABLE_TABLE_SET_DISABLED_TABLE_STATE; DisableTableProcedure table=TestTable.testCheckAndPutHooks; resume parent processing. 2017-10-03 21:50:55,601 INFO [ProcExecWrkr-19] procedure2.ProcedureExecutor(1246): Finished pid=51, ppid=50, state=SUCCESS; UnassignProcedure table=TestTable.testCheckAndPutHooks, region=eac18e9cf3db3c83e2236ac664f87506, server=cn012.l42scl.hortonworks.com,37573,1507067090360 in 692msec 2017-10-03 21:50:55,601 DEBUG [ProcExecWrkr-21] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testCheckAndPutHooks","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067455601}]}} 2017-10-03 21:50:55,610 INFO [ProcExecWrkr-21] hbase.MetaTableAccessor(1770): Updated table TestTable.testCheckAndPutHooks state to DISABLED in META 2017-10-03 21:50:55,611 INFO [ProcExecWrkr-21] procedure.DisableTableProcedure(285): Disabled table, TestTable.testCheckAndPutHooks, is completed. 2017-10-03 21:50:55,859 INFO [ProcExecWrkr-21] procedure2.ProcedureExecutor(1246): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestTable.testCheckAndPutHooks in 1.5140sec 2017-10-03 21:50:55,986 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 266829ms 2017-10-03 21:50:55,986 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 273995ms 2017-10-03 21:50:56,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=50 2017-10-03 21:50:56,525 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DISABLE, Table Name: default:TestTable.testCheckAndPutHooks completed 2017-10-03 21:50:56,526 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$5(2109): Client=hbase//172.18.128.12 delete TestTable.testCheckAndPutHooks 2017-10-03 21:50:56,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=52, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestTable.testCheckAndPutHooks 2017-10-03 21:50:56,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=52 2017-10-03 21:50:56,677 DEBUG [ProcExecWrkr-23] procedure.DeleteTableProcedure(99): Waiting for 'TestTable.testCheckAndPutHooks' regions in transition 2017-10-03 21:50:56,754 DEBUG [ProcExecWrkr-23] procedure.DeleteTableProcedure(110): delete 'TestTable.testCheckAndPutHooks' regions from META 2017-10-03 21:50:56,755 DEBUG [ProcExecWrkr-23] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067456754}]},"ts":9223372036854775807} 2017-10-03 21:50:56,760 INFO [ProcExecWrkr-23] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:50:56,760 DEBUG [ProcExecWrkr-23] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => eac18e9cf3db3c83e2236ac664f87506, NAME => 'TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:50:56,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=52 2017-10-03 21:50:56,871 DEBUG [ProcExecWrkr-23] procedure.DeleteTableProcedure(115): delete 'TestTable.testCheckAndPutHooks' from filesystem 2017-10-03 21:50:56,874 DEBUG [ProcExecWrkr-23] procedure.DeleteTableProcedure(315): Archiving region TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506. from FS 2017-10-03 21:50:56,874 DEBUG [ProcExecWrkr-23] backup.HFileArchiver(114): ARCHIVING hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506 2017-10-03 21:50:56,880 DEBUG [ProcExecWrkr-23] backup.HFileArchiver(155): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/recovered.edits] 2017-10-03 21:50:56,888 DEBUG [ProcExecWrkr-23] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/a/f99d6d663d8e4fd9bae8c78a9cb3d684, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/a/f99d6d663d8e4fd9bae8c78a9cb3d684 2017-10-03 21:50:56,897 DEBUG [ProcExecWrkr-23] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/recovered.edits/9.seqid, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506/recovered.edits/9.seqid 2017-10-03 21:50:56,898 INFO [IPC Server handler 7 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741866_1042 127.0.0.1:41361 2017-10-03 21:50:56,899 DEBUG [ProcExecWrkr-23] backup.HFileArchiver(466): Deleted hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testCheckAndPutHooks/eac18e9cf3db3c83e2236ac664f87506 2017-10-03 21:50:56,901 DEBUG [ProcExecWrkr-23] procedure.DeleteTableProcedure(319): Table 'TestTable.testCheckAndPutHooks' archived! 2017-10-03 21:50:56,902 INFO [IPC Server handler 1 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741865_1041 127.0.0.1:41361 2017-10-03 21:50:56,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 22781ms 2017-10-03 21:50:56,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 297371ms 2017-10-03 21:50:56,937 DEBUG [ProcExecWrkr-23] procedure.DeleteTableProcedure(121): delete 'TestTable.testCheckAndPutHooks' descriptor 2017-10-03 21:50:56,937 DEBUG [ProcExecWrkr-23] procedure.DeleteTableProcedure(399): Removing 'TestTable.testCheckAndPutHooks' descriptor. 2017-10-03 21:50:56,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=52 2017-10-03 21:50:57,056 DEBUG [ProcExecWrkr-23] procedure.DeleteTableProcedure(126): delete 'TestTable.testCheckAndPutHooks' assignment state 2017-10-03 21:50:57,056 DEBUG [ProcExecWrkr-23] procedure.DeleteTableProcedure(389): Removing 'TestTable.testCheckAndPutHooks' from region states. 2017-10-03 21:50:57,058 DEBUG [ProcExecWrkr-23] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067457057}]},"ts":9223372036854775807} 2017-10-03 21:50:57,066 INFO [ProcExecWrkr-23] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:50:57,066 DEBUG [ProcExecWrkr-23] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => eac18e9cf3db3c83e2236ac664f87506, NAME => 'TestTable.testCheckAndPutHooks,,1507067451928.eac18e9cf3db3c83e2236ac664f87506.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:50:57,066 DEBUG [ProcExecWrkr-23] procedure.DeleteTableProcedure(393): Marking 'TestTable.testCheckAndPutHooks' as deleted. 2017-10-03 21:50:57,067 DEBUG [ProcExecWrkr-23] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"TestTable.testCheckAndPutHooks","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":1507067457066}]},"ts":9223372036854775807} 2017-10-03 21:50:57,070 INFO [ProcExecWrkr-23] hbase.MetaTableAccessor(1796): Deleted table TestTable.testCheckAndPutHooks state from META 2017-10-03 21:50:57,185 DEBUG [ProcExecWrkr-23] procedure.DeleteTableProcedure(132): delete 'TestTable.testCheckAndPutHooks' completed 2017-10-03 21:50:57,272 INFO [ProcExecWrkr-23] procedure2.ProcedureExecutor(1246): Finished pid=52, state=SUCCESS; DeleteTableProcedure table=TestTable.testCheckAndPutHooks in 658msec 2017-10-03 21:50:57,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=52 2017-10-03 21:50:57,282 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DELETE, Table Name: default:TestTable.testCheckAndPutHooks completed 2017-10-03 21:50:57,329 INFO [main] hbase.ResourceChecker(172): after: coprocessor.TestRegionObserverInterface#testCheckAndPutHooks Thread=294 (was 289) - Thread LEAK? -, OpenFileDescriptor=1246 (was 1246), MaxFileDescriptor=32000 (was 32000), SystemLoadAverage=389 (was 388) - SystemLoadAverage LEAK? -, ProcessCount=347 (was 347), AvailableMemoryMB=48637 (was 48716) 2017-10-03 21:50:57,329 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1246 is superior to 1024 2017-10-03 21:50:57,364 INFO [main] hbase.ResourceChecker(148): before: coprocessor.TestRegionObserverInterface#testRowMutation Thread=294, OpenFileDescriptor=1246, MaxFileDescriptor=32000, SystemLoadAverage=389, ProcessCount=347, AvailableMemoryMB=48637 2017-10-03 21:50:57,364 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1246 is superior to 1024 2017-10-03 21:50:57,371 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$3(1748): Client=hbase//172.18.128.12 create 'TestTable.testRowMutation', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2017-10-03 21:50:57,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=53, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestTable.testRowMutation 2017-10-03 21:50:57,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=53 2017-10-03 21:50:57,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=53 2017-10-03 21:50:57,744 INFO [IPC Server handler 7 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741868_1044{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|RBW]]} size 1440 2017-10-03 21:50:57,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=53 2017-10-03 21:50:58,010 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 90628ms 2017-10-03 21:50:58,010 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 234935ms 2017-10-03 21:50:58,150 DEBUG [ProcExecWrkr-22] util.FSTableDescriptors(728): Wrote descriptor into: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRowMutation/.tabledesc/.tableinfo.0000000001 2017-10-03 21:50:58,153 INFO [RegionOpenAndInitThread-TestTable.testRowMutation-1] regionserver.HRegion(6477): creating HRegion TestTable.testRowMutation HTD == 'TestTable.testRowMutation', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp Table name == TestTable.testRowMutation 2017-10-03 21:50:58,167 INFO [IPC Server handler 6 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741869_1045{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 0 2017-10-03 21:50:58,168 DEBUG [RegionOpenAndInitThread-TestTable.testRowMutation-1] regionserver.HRegion(798): Instantiated TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. 2017-10-03 21:50:58,169 DEBUG [RegionOpenAndInitThread-TestTable.testRowMutation-1] regionserver.HRegion(1560): Closing TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8.: disabling compactions & flushes 2017-10-03 21:50:58,169 DEBUG [RegionOpenAndInitThread-TestTable.testRowMutation-1] regionserver.HRegion(1599): Updates disabled for region TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. 2017-10-03 21:50:58,169 INFO [RegionOpenAndInitThread-TestTable.testRowMutation-1] regionserver.HRegion(1712): Closed TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. 2017-10-03 21:50:58,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=53 2017-10-03 21:50:58,265 DEBUG [ProcExecWrkr-22] hbase.MetaTableAccessor(1417): Put{"totalColumns":1,"row":"TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8.","families":{"info":[{"qualifier":"regioninfo","vlen":59,"tag":[],"timestamp":9223372036854775807}]}} 2017-10-03 21:50:58,270 INFO [ProcExecWrkr-22] hbase.MetaTableAccessor(1614): Added 1 2017-10-03 21:50:58,359 DEBUG [ProcExecWrkr-22] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testRowMutation","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067458358}]}} 2017-10-03 21:50:58,364 INFO [ProcExecWrkr-22] hbase.MetaTableAccessor(1770): Updated table TestTable.testRowMutation state to ENABLING in META 2017-10-03 21:50:58,364 INFO [ProcExecWrkr-22] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=54, ppid=53, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testRowMutation, region=36282319f03be3ad27e61d47d16604b8}] 2017-10-03 21:50:58,403 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@70efb718] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:41361 to delete [blk_1073741865_1041, blk_1073741866_1042] 2017-10-03 21:50:58,448 INFO [ProcExecWrkr-1] procedure.MasterProcedureScheduler(814): pid=54, ppid=53, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testRowMutation, region=36282319f03be3ad27e61d47d16604b8 TestTable.testRowMutation TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. 2017-10-03 21:50:58,451 INFO [ProcExecWrkr-1] assignment.AssignProcedure(204): Start pid=54, ppid=53, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testRowMutation, region=36282319f03be3ad27e61d47d16604b8; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2017-10-03 21:50:58,603 INFO [ProcExecWrkr-24] assignment.RegionStateStore(186): pid=54 updating hbase:meta row=TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8., regionState=OPENING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:58,613 INFO [ProcExecWrkr-24] assignment.RegionTransitionProcedure(213): Dispatch pid=54, ppid=53, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testRowMutation, region=36282319f03be3ad27e61d47d16604b8; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:58,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=53 2017-10-03 21:50:58,764 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:50:58,766 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1916): Open TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. 2017-10-03 21:50:58,773 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(6654): Opening region: {ENCODED => 36282319f03be3ad27e61d47d16604b8, NAME => 'TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8.', STARTKEY => '', ENDKEY => ''} 2017-10-03 21:50:58,774 INFO [RS_OPEN_REGION-cn012:37573-1] coprocessor.CoprocessorHost(164): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2017-10-03 21:50:58,774 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table TestTable.testRowMutation 36282319f03be3ad27e61d47d16604b8 2017-10-03 21:50:58,774 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(798): Instantiated TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. 2017-10-03 21:50:58,782 DEBUG [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/a 2017-10-03 21:50:58,783 DEBUG [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/a 2017-10-03 21:50:58,783 INFO [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:50:58,783 INFO [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:50:58,784 INFO [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] hfile.CacheConfig(258): Created cacheConfig for a: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:50:58,784 INFO [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:50:58,789 DEBUG [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/b 2017-10-03 21:50:58,789 DEBUG [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/b 2017-10-03 21:50:58,790 INFO [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:50:58,790 INFO [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:50:58,790 INFO [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] hfile.CacheConfig(258): Created cacheConfig for b: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:50:58,790 INFO [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:50:58,795 DEBUG [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/c 2017-10-03 21:50:58,795 DEBUG [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/c 2017-10-03 21:50:58,795 INFO [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:50:58,796 INFO [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:50:58,796 INFO [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] hfile.CacheConfig(258): Created cacheConfig for c: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:50:58,796 INFO [StoreOpener-36282319f03be3ad27e61d47d16604b8-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:50:58,798 INFO [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(1030): Setting FlushNonSloppyStoresFirstPolicy for the region=TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. 2017-10-03 21:50:58,800 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(4121): Found 0 recovered edits file(s) under hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8 2017-10-03 21:50:58,804 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table TestTable.testRowMutation, use config (44739242) instead 2017-10-03 21:50:58,809 DEBUG [RS_OPEN_REGION-cn012:37573-1] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2017-10-03 21:50:58,810 INFO [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(957): Onlined 36282319f03be3ad27e61d47d16604b8; next sequenceid=2 2017-10-03 21:50:58,812 INFO [PostOpenDeployTasks:36282319f03be3ad27e61d47d16604b8] regionserver.HRegionServer(2170): Post open deploy tasks for TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. 2017-10-03 21:50:58,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=54, ppid=53, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testRowMutation, region=36282319f03be3ad27e61d47d16604b8; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:58,817 INFO [ProcExecWrkr-2] assignment.RegionStateStore(186): pid=54 updating hbase:meta row=TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8., regionState=OPEN, openSeqNum=2, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:58,818 DEBUG [PostOpenDeployTasks:36282319f03be3ad27e61d47d16604b8] regionserver.HRegionServer(2197): Finished post open deploy task for TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. 2017-10-03 21:50:58,819 DEBUG [RS_OPEN_REGION-cn012:37573-1] handler.OpenRegionHandler(127): Opened TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. on cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:50:58,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 39248ms 2017-10-03 21:50:58,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 197567ms 2017-10-03 21:50:58,925 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=54, ppid=53, state=SUCCESS; AssignProcedure table=TestTable.testRowMutation, region=36282319f03be3ad27e61d47d16604b8 2017-10-03 21:50:59,005 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=53, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=TestTable.testRowMutation; resume parent processing. 2017-10-03 21:50:59,006 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1246): Finished pid=54, ppid=53, state=SUCCESS; AssignProcedure table=TestTable.testRowMutation, region=36282319f03be3ad27e61d47d16604b8 in 459msec 2017-10-03 21:50:59,006 DEBUG [ProcExecWrkr-6] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testRowMutation","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067459005}]}} 2017-10-03 21:50:59,010 INFO [ProcExecWrkr-6] hbase.MetaTableAccessor(1770): Updated table TestTable.testRowMutation state to ENABLED in META 2017-10-03 21:50:59,289 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(135): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor 2017-10-03 21:50:59,289 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(139): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor Metrics about HBase RegionObservers 2017-10-03 21:50:59,303 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1246): Finished pid=53, state=SUCCESS; CreateTableProcedure table=TestTable.testRowMutation in 1.8400sec 2017-10-03 21:50:59,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=53 2017-10-03 21:50:59,733 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: CREATE, Table Name: default:TestTable.testRowMutation completed 2017-10-03 21:50:59,734 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:50:59,743 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:50:59,761 INFO [Time-limited test] client.HBaseAdmin$15(919): Started disable of TestTable.testRowMutation 2017-10-03 21:50:59,761 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$11(2331): Client=hbase//172.18.128.12 disable TestTable.testRowMutation 2017-10-03 21:50:59,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=55, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestTable.testRowMutation 2017-10-03 21:50:59,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=55 2017-10-03 21:50:59,985 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 19389ms 2017-10-03 21:50:59,985 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 267802ms 2017-10-03 21:51:00,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=55 2017-10-03 21:51:00,123 DEBUG [ProcExecWrkr-4] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testRowMutation","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067460122}]}} 2017-10-03 21:51:00,130 INFO [ProcExecWrkr-4] hbase.MetaTableAccessor(1770): Updated table TestTable.testRowMutation state to DISABLING in META 2017-10-03 21:51:00,223 INFO [ProcExecWrkr-4] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testRowMutation, region=36282319f03be3ad27e61d47d16604b8, server=cn012.l42scl.hortonworks.com,37573,1507067090360}] 2017-10-03 21:51:00,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=55 2017-10-03 21:51:00,301 INFO [ProcExecWrkr-5] procedure.MasterProcedureScheduler(814): pid=56, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testRowMutation, region=36282319f03be3ad27e61d47d16604b8, server=cn012.l42scl.hortonworks.com,37573,1507067090360 TestTable.testRowMutation TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. 2017-10-03 21:51:00,302 INFO [ProcExecWrkr-5] assignment.RegionStateStore(186): pid=56 updating hbase:meta row=TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8., regionState=CLOSING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:00,308 INFO [ProcExecWrkr-5] assignment.RegionTransitionProcedure(213): Dispatch pid=56, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testRowMutation, region=36282319f03be3ad27e61d47d16604b8, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:00,458 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:51:00,460 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1520): Close 36282319f03be3ad27e61d47d16604b8 without moving 2017-10-03 21:51:00,460 DEBUG [RS_CLOSE_REGION-cn012:37573-2] handler.CloseRegionHandler(91): Processing close of TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. 2017-10-03 21:51:00,461 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(1560): Closing TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8.: disabling compactions & flushes 2017-10-03 21:51:00,462 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(1599): Updates disabled for region TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. 2017-10-03 21:51:00,462 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(2535): Flushing 3/3 column families, memstore=177 B 2017-10-03 21:51:00,462 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8.store: a 2017-10-03 21:51:00,463 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:51:00,463 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8.store: b 2017-10-03 21:51:00,463 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:51:00,463 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8.store: c 2017-10-03 21:51:00,463 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:51:00,480 INFO [IPC Server handler 7 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741870_1046{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|RBW]]} size 4668 2017-10-03 21:51:00,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=55 2017-10-03 21:51:00,863 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2017-10-03 21:51:00,882 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=5, memsize=59, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/.tmp/a/35a4c671bf34488cba824cac86e13f43 2017-10-03 21:51:00,907 INFO [IPC Server handler 5 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741871_1047{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 4668 2017-10-03 21:51:00,917 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 7529ms 2017-10-03 21:51:00,918 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 160055ms 2017-10-03 21:51:01,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=55 2017-10-03 21:51:01,309 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=5, memsize=59, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/.tmp/b/4fc8b73afc2e4301934c64dcb2411447 2017-10-03 21:51:01,344 INFO [IPC Server handler 7 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741872_1048{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|RBW]]} size 4668 2017-10-03 21:51:01,746 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=5, memsize=59, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/.tmp/c/a011b6902d0e4fd6aaef3940584ab599 2017-10-03 21:51:01,759 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/.tmp/a/35a4c671bf34488cba824cac86e13f43 as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/a/35a4c671bf34488cba824cac86e13f43 2017-10-03 21:51:01,768 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/a/35a4c671bf34488cba824cac86e13f43, entries=1, sequenceid=5, filesize=4.6 K 2017-10-03 21:51:01,770 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/.tmp/b/4fc8b73afc2e4301934c64dcb2411447 as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/b/4fc8b73afc2e4301934c64dcb2411447 2017-10-03 21:51:01,776 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/b/4fc8b73afc2e4301934c64dcb2411447, entries=1, sequenceid=5, filesize=4.6 K 2017-10-03 21:51:01,778 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/.tmp/c/a011b6902d0e4fd6aaef3940584ab599 as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/c/a011b6902d0e4fd6aaef3940584ab599 2017-10-03 21:51:01,785 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/c/a011b6902d0e4fd6aaef3940584ab599, entries=1, sequenceid=5, filesize=4.6 K 2017-10-03 21:51:01,787 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(2731): Finished memstore flush of ~177 B/177, currentsize=0 B/0 for region TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. in 1325ms, sequenceid=5, compaction requested=false 2017-10-03 21:51:01,789 INFO [StoreCloserThread-TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8.-1] regionserver.HStore(926): Closed a 2017-10-03 21:51:01,790 INFO [StoreCloserThread-TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8.-1] regionserver.HStore(926): Closed b 2017-10-03 21:51:01,791 INFO [StoreCloserThread-TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8.-1] regionserver.HStore(926): Closed c 2017-10-03 21:51:01,798 DEBUG [RS_CLOSE_REGION-cn012:37573-2] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/recovered.edits/8.seqid, newSeqId=8, maxSeqId=2 2017-10-03 21:51:01,799 DEBUG [RS_CLOSE_REGION-cn012:37573-2] coprocessor.CoprocessorHost(291): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2017-10-03 21:51:01,799 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(1712): Closed TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. 2017-10-03 21:51:01,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report CLOSED seqId=-1, pid=56, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testRowMutation, region=36282319f03be3ad27e61d47d16604b8, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:01,801 DEBUG [RS_CLOSE_REGION-cn012:37573-2] handler.CloseRegionHandler(123): Closed TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. 2017-10-03 21:51:01,801 INFO [ProcExecWrkr-3] assignment.RegionStateStore(186): pid=56 updating hbase:meta row=TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8., regionState=CLOSED 2017-10-03 21:51:01,847 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=56, ppid=55, state=SUCCESS; UnassignProcedure table=TestTable.testRowMutation, region=36282319f03be3ad27e61d47d16604b8, server=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:01,888 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=55, state=RUNNABLE:DISABLE_TABLE_SET_DISABLED_TABLE_STATE; DisableTableProcedure table=TestTable.testRowMutation; resume parent processing. 2017-10-03 21:51:01,889 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1246): Finished pid=56, ppid=55, state=SUCCESS; UnassignProcedure table=TestTable.testRowMutation, region=36282319f03be3ad27e61d47d16604b8, server=cn012.l42scl.hortonworks.com,37573,1507067090360 in 1.5830sec 2017-10-03 21:51:01,889 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testRowMutation","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067461889}]}} 2017-10-03 21:51:01,893 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1770): Updated table TestTable.testRowMutation state to DISABLED in META 2017-10-03 21:51:01,893 INFO [ProcExecWrkr-7] procedure.DisableTableProcedure(285): Disabled table, TestTable.testRowMutation, is completed. 2017-10-03 21:51:01,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 42483ms 2017-10-03 21:51:01,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 80328ms 2017-10-03 21:51:02,022 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1246): Finished pid=55, state=SUCCESS; DisableTableProcedure table=TestTable.testRowMutation in 2.1680sec 2017-10-03 21:51:02,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=55 2017-10-03 21:51:02,080 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DISABLE, Table Name: default:TestTable.testRowMutation completed 2017-10-03 21:51:02,082 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$5(2109): Client=hbase//172.18.128.12 delete TestTable.testRowMutation 2017-10-03 21:51:02,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=57, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestTable.testRowMutation 2017-10-03 21:51:02,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=57 2017-10-03 21:51:02,237 DEBUG [ProcExecWrkr-10] procedure.DeleteTableProcedure(99): Waiting for 'TestTable.testRowMutation' regions in transition 2017-10-03 21:51:02,289 DEBUG [ProcExecWrkr-10] procedure.DeleteTableProcedure(110): delete 'TestTable.testRowMutation' regions from META 2017-10-03 21:51:02,289 DEBUG [ProcExecWrkr-10] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067462289}]},"ts":9223372036854775807} 2017-10-03 21:51:02,294 INFO [ProcExecWrkr-10] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:51:02,294 DEBUG [ProcExecWrkr-10] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => 36282319f03be3ad27e61d47d16604b8, NAME => 'TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:51:02,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=57 2017-10-03 21:51:02,347 DEBUG [ProcExecWrkr-10] procedure.DeleteTableProcedure(115): delete 'TestTable.testRowMutation' from filesystem 2017-10-03 21:51:02,351 DEBUG [ProcExecWrkr-10] procedure.DeleteTableProcedure(315): Archiving region TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8. from FS 2017-10-03 21:51:02,351 DEBUG [ProcExecWrkr-10] backup.HFileArchiver(114): ARCHIVING hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8 2017-10-03 21:51:02,356 DEBUG [ProcExecWrkr-10] backup.HFileArchiver(155): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/recovered.edits] 2017-10-03 21:51:02,365 DEBUG [ProcExecWrkr-10] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/a/35a4c671bf34488cba824cac86e13f43, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/a/35a4c671bf34488cba824cac86e13f43 2017-10-03 21:51:02,371 DEBUG [ProcExecWrkr-10] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/b/4fc8b73afc2e4301934c64dcb2411447, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/b/4fc8b73afc2e4301934c64dcb2411447 2017-10-03 21:51:02,376 DEBUG [ProcExecWrkr-10] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/c/a011b6902d0e4fd6aaef3940584ab599, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/c/a011b6902d0e4fd6aaef3940584ab599 2017-10-03 21:51:02,381 DEBUG [ProcExecWrkr-10] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/recovered.edits/8.seqid, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8/recovered.edits/8.seqid 2017-10-03 21:51:02,382 INFO [IPC Server handler 8 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741869_1045 127.0.0.1:41361 2017-10-03 21:51:02,383 DEBUG [ProcExecWrkr-10] backup.HFileArchiver(466): Deleted hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRowMutation/36282319f03be3ad27e61d47d16604b8 2017-10-03 21:51:02,383 DEBUG [ProcExecWrkr-10] procedure.DeleteTableProcedure(319): Table 'TestTable.testRowMutation' archived! 2017-10-03 21:51:02,384 INFO [IPC Server handler 2 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741868_1044 127.0.0.1:41361 2017-10-03 21:51:02,430 DEBUG [ProcExecWrkr-10] procedure.DeleteTableProcedure(121): delete 'TestTable.testRowMutation' descriptor 2017-10-03 21:51:02,430 DEBUG [ProcExecWrkr-10] procedure.DeleteTableProcedure(399): Removing 'TestTable.testRowMutation' descriptor. 2017-10-03 21:51:02,534 DEBUG [ProcExecWrkr-10] procedure.DeleteTableProcedure(126): delete 'TestTable.testRowMutation' assignment state 2017-10-03 21:51:02,535 DEBUG [ProcExecWrkr-10] procedure.DeleteTableProcedure(389): Removing 'TestTable.testRowMutation' from region states. 2017-10-03 21:51:02,535 DEBUG [ProcExecWrkr-10] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067462535}]},"ts":9223372036854775807} 2017-10-03 21:51:02,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=57 2017-10-03 21:51:02,539 INFO [ProcExecWrkr-10] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:51:02,540 DEBUG [ProcExecWrkr-10] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => 36282319f03be3ad27e61d47d16604b8, NAME => 'TestTable.testRowMutation,,1507067457370.36282319f03be3ad27e61d47d16604b8.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:51:02,540 DEBUG [ProcExecWrkr-10] procedure.DeleteTableProcedure(393): Marking 'TestTable.testRowMutation' as deleted. 2017-10-03 21:51:02,540 DEBUG [ProcExecWrkr-10] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"TestTable.testRowMutation","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":1507067462540}]},"ts":9223372036854775807} 2017-10-03 21:51:02,543 INFO [ProcExecWrkr-10] hbase.MetaTableAccessor(1796): Deleted table TestTable.testRowMutation state from META 2017-10-03 21:51:02,627 DEBUG [ProcExecWrkr-10] procedure.DeleteTableProcedure(132): delete 'TestTable.testRowMutation' completed 2017-10-03 21:51:02,697 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1246): Finished pid=57, state=SUCCESS; DeleteTableProcedure table=TestTable.testRowMutation in 545msec 2017-10-03 21:51:02,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=57 2017-10-03 21:51:02,840 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DELETE, Table Name: default:TestTable.testRowMutation completed 2017-10-03 21:51:02,886 INFO [main] hbase.ResourceChecker(172): after: coprocessor.TestRegionObserverInterface#testRowMutation Thread=300 (was 294) - Thread LEAK? -, OpenFileDescriptor=1246 (was 1246), MaxFileDescriptor=32000 (was 32000), SystemLoadAverage=398 (was 389) - SystemLoadAverage LEAK? -, ProcessCount=349 (was 347) - ProcessCount LEAK? -, AvailableMemoryMB=48583 (was 48637) 2017-10-03 21:51:02,886 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1246 is superior to 1024 2017-10-03 21:51:02,918 INFO [main] hbase.ResourceChecker(148): before: coprocessor.TestRegionObserverInterface#testRegionObserver Thread=300, OpenFileDescriptor=1246, MaxFileDescriptor=32000, SystemLoadAverage=398, ProcessCount=349, AvailableMemoryMB=48583 2017-10-03 21:51:02,918 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1246 is superior to 1024 2017-10-03 21:51:02,919 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 56509ms 2017-10-03 21:51:02,921 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 94062ms 2017-10-03 21:51:02,924 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$3(1748): Client=hbase//172.18.128.12 create 'TestTable.testRegionObserver', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2017-10-03 21:51:03,330 WARN [DataXceiver for client DFSClient_NONMAPREDUCE_-174551704_1 at /127.0.0.1:58812 [Receiving block BP-608549805-172.18.128.12-1507067084534:blk_1073741829_1005]] datanode.BlockReceiver(391): Slow flushOrSync took 305ms (threshold=300ms), isSync:true, flushTotalNanos=8398ns 2017-10-03 21:51:03,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=58, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestTable.testRegionObserver 2017-10-03 21:51:03,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=58 2017-10-03 21:51:03,398 INFO [IPC Server handler 1 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741873_1049{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|RBW]]} size 1443 2017-10-03 21:51:03,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=58 2017-10-03 21:51:03,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=58 2017-10-03 21:51:03,803 DEBUG [ProcExecWrkr-8] util.FSTableDescriptors(728): Wrote descriptor into: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRegionObserver/.tabledesc/.tableinfo.0000000001 2017-10-03 21:51:03,806 INFO [RegionOpenAndInitThread-TestTable.testRegionObserver-1] regionserver.HRegion(6477): creating HRegion TestTable.testRegionObserver HTD == 'TestTable.testRegionObserver', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp Table name == TestTable.testRegionObserver 2017-10-03 21:51:03,819 INFO [IPC Server handler 7 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741874_1050{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|RBW]]} size 0 2017-10-03 21:51:03,821 DEBUG [RegionOpenAndInitThread-TestTable.testRegionObserver-1] regionserver.HRegion(798): Instantiated TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. 2017-10-03 21:51:03,821 DEBUG [RegionOpenAndInitThread-TestTable.testRegionObserver-1] regionserver.HRegion(1560): Closing TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21.: disabling compactions & flushes 2017-10-03 21:51:03,821 DEBUG [RegionOpenAndInitThread-TestTable.testRegionObserver-1] regionserver.HRegion(1599): Updates disabled for region TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. 2017-10-03 21:51:03,821 INFO [RegionOpenAndInitThread-TestTable.testRegionObserver-1] regionserver.HRegion(1712): Closed TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. 2017-10-03 21:51:03,896 DEBUG [ProcExecWrkr-8] hbase.MetaTableAccessor(1417): Put{"totalColumns":1,"row":"TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21.","families":{"info":[{"qualifier":"regioninfo","vlen":62,"tag":[],"timestamp":9223372036854775807}]}} 2017-10-03 21:51:03,902 INFO [ProcExecWrkr-8] hbase.MetaTableAccessor(1614): Added 1 2017-10-03 21:51:03,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 11260ms 2017-10-03 21:51:03,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 84531ms 2017-10-03 21:51:03,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=58 2017-10-03 21:51:03,947 DEBUG [ProcExecWrkr-8] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testRegionObserver","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067463947}]}} 2017-10-03 21:51:03,951 INFO [ProcExecWrkr-8] hbase.MetaTableAccessor(1770): Updated table TestTable.testRegionObserver state to ENABLING in META 2017-10-03 21:51:03,952 INFO [ProcExecWrkr-8] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testRegionObserver, region=e7270d04ea8b4e52e1e1ccdbe7082c21}] 2017-10-03 21:51:04,019 INFO [ProcExecWrkr-8] procedure.MasterProcedureScheduler(814): pid=59, ppid=58, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testRegionObserver, region=e7270d04ea8b4e52e1e1ccdbe7082c21 TestTable.testRegionObserver TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. 2017-10-03 21:51:04,022 INFO [ProcExecWrkr-8] assignment.AssignProcedure(204): Start pid=59, ppid=58, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testRegionObserver, region=e7270d04ea8b4e52e1e1ccdbe7082c21; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2017-10-03 21:51:04,176 INFO [ProcExecWrkr-12] assignment.RegionStateStore(186): pid=59 updating hbase:meta row=TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21., regionState=OPENING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:04,191 INFO [ProcExecWrkr-12] assignment.RegionTransitionProcedure(213): Dispatch pid=59, ppid=58, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testRegionObserver, region=e7270d04ea8b4e52e1e1ccdbe7082c21; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:04,341 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:51:04,343 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1916): Open TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. 2017-10-03 21:51:04,349 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(6654): Opening region: {ENCODED => e7270d04ea8b4e52e1e1ccdbe7082c21, NAME => 'TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21.', STARTKEY => '', ENDKEY => ''} 2017-10-03 21:51:04,350 INFO [RS_OPEN_REGION-cn012:37573-2] coprocessor.CoprocessorHost(164): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2017-10-03 21:51:04,350 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table TestTable.testRegionObserver e7270d04ea8b4e52e1e1ccdbe7082c21 2017-10-03 21:51:04,351 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(798): Instantiated TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. 2017-10-03 21:51:04,357 DEBUG [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/a 2017-10-03 21:51:04,357 DEBUG [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/a 2017-10-03 21:51:04,358 INFO [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:51:04,358 INFO [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:51:04,359 INFO [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] hfile.CacheConfig(258): Created cacheConfig for a: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:51:04,359 INFO [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:51:04,363 DEBUG [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/b 2017-10-03 21:51:04,363 DEBUG [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/b 2017-10-03 21:51:04,363 INFO [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:51:04,363 INFO [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:51:04,363 INFO [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] hfile.CacheConfig(258): Created cacheConfig for b: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:51:04,363 INFO [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:51:04,366 DEBUG [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/c 2017-10-03 21:51:04,367 DEBUG [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/c 2017-10-03 21:51:04,367 INFO [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:51:04,367 INFO [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:51:04,367 INFO [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] hfile.CacheConfig(258): Created cacheConfig for c: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:51:04,367 INFO [StoreOpener-e7270d04ea8b4e52e1e1ccdbe7082c21-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:51:04,368 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(1030): Setting FlushNonSloppyStoresFirstPolicy for the region=TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. 2017-10-03 21:51:04,369 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(4121): Found 0 recovered edits file(s) under hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21 2017-10-03 21:51:04,371 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table TestTable.testRegionObserver, use config (44739242) instead 2017-10-03 21:51:04,374 DEBUG [RS_OPEN_REGION-cn012:37573-2] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2017-10-03 21:51:04,375 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(957): Onlined e7270d04ea8b4e52e1e1ccdbe7082c21; next sequenceid=2 2017-10-03 21:51:04,376 INFO [PostOpenDeployTasks:e7270d04ea8b4e52e1e1ccdbe7082c21] regionserver.HRegionServer(2170): Post open deploy tasks for TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. 2017-10-03 21:51:04,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=59, ppid=58, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testRegionObserver, region=e7270d04ea8b4e52e1e1ccdbe7082c21; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:04,378 INFO [ProcExecWrkr-15] assignment.RegionStateStore(186): pid=59 updating hbase:meta row=TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21., regionState=OPEN, openSeqNum=2, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:04,379 DEBUG [PostOpenDeployTasks:e7270d04ea8b4e52e1e1ccdbe7082c21] regionserver.HRegionServer(2197): Finished post open deploy task for TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. 2017-10-03 21:51:04,379 DEBUG [RS_OPEN_REGION-cn012:37573-2] handler.OpenRegionHandler(127): Opened TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. on cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:04,386 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2017-10-03 21:51:04,404 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@70efb718] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:41361 to delete [blk_1073741868_1044, blk_1073741869_1045] 2017-10-03 21:51:04,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=58 2017-10-03 21:51:04,443 INFO [ProcExecWrkr-15] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=59, ppid=58, state=SUCCESS; AssignProcedure table=TestTable.testRegionObserver, region=e7270d04ea8b4e52e1e1ccdbe7082c21 2017-10-03 21:51:04,529 INFO [ProcExecWrkr-15] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=58, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=TestTable.testRegionObserver; resume parent processing. 2017-10-03 21:51:04,529 INFO [ProcExecWrkr-15] procedure2.ProcedureExecutor(1246): Finished pid=59, ppid=58, state=SUCCESS; AssignProcedure table=TestTable.testRegionObserver, region=e7270d04ea8b4e52e1e1ccdbe7082c21 in 430msec 2017-10-03 21:51:04,529 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testRegionObserver","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067464529}]}} 2017-10-03 21:51:04,533 INFO [ProcExecWrkr-14] hbase.MetaTableAccessor(1770): Updated table TestTable.testRegionObserver state to ENABLED in META 2017-10-03 21:51:04,706 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1246): Finished pid=58, state=SUCCESS; CreateTableProcedure table=TestTable.testRegionObserver in 1.7150sec 2017-10-03 21:51:04,988 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 83212ms 2017-10-03 21:51:04,988 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 120449ms 2017-10-03 21:51:05,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=58 2017-10-03 21:51:05,445 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: CREATE, Table Name: default:TestTable.testRegionObserver completed 2017-10-03 21:51:05,445 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:51:05,452 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:51:05,467 INFO [Time-limited test] client.HBaseAdmin$15(919): Started disable of TestTable.testRegionObserver 2017-10-03 21:51:05,468 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$11(2331): Client=hbase//172.18.128.12 disable TestTable.testRegionObserver 2017-10-03 21:51:05,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=60, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestTable.testRegionObserver 2017-10-03 21:51:05,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=60 2017-10-03 21:51:05,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=60 2017-10-03 21:51:05,831 DEBUG [ProcExecWrkr-13] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testRegionObserver","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067465830}]}} 2017-10-03 21:51:05,836 INFO [ProcExecWrkr-13] hbase.MetaTableAccessor(1770): Updated table TestTable.testRegionObserver state to DISABLING in META 2017-10-03 21:51:05,900 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=61, ppid=60, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testRegionObserver, region=e7270d04ea8b4e52e1e1ccdbe7082c21, server=cn012.l42scl.hortonworks.com,37573,1507067090360}] 2017-10-03 21:51:05,947 INFO [ProcExecWrkr-13] procedure.MasterProcedureScheduler(814): pid=61, ppid=60, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testRegionObserver, region=e7270d04ea8b4e52e1e1ccdbe7082c21, server=cn012.l42scl.hortonworks.com,37573,1507067090360 TestTable.testRegionObserver TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. 2017-10-03 21:51:05,947 INFO [ProcExecWrkr-13] assignment.RegionStateStore(186): pid=61 updating hbase:meta row=TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21., regionState=CLOSING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:05,952 INFO [ProcExecWrkr-13] assignment.RegionTransitionProcedure(213): Dispatch pid=61, ppid=60, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testRegionObserver, region=e7270d04ea8b4e52e1e1ccdbe7082c21, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:05,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=60 2017-10-03 21:51:05,969 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 80679ms 2017-10-03 21:51:05,969 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 209950ms 2017-10-03 21:51:06,102 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:51:06,104 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1520): Close e7270d04ea8b4e52e1e1ccdbe7082c21 without moving 2017-10-03 21:51:06,105 DEBUG [RS_CLOSE_REGION-cn012:37573-0] handler.CloseRegionHandler(91): Processing close of TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. 2017-10-03 21:51:06,106 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(1560): Closing TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21.: disabling compactions & flushes 2017-10-03 21:51:06,106 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(1599): Updates disabled for region TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. 2017-10-03 21:51:06,106 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(2535): Flushing 3/3 column families, memstore=177 B 2017-10-03 21:51:06,107 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21.store: a 2017-10-03 21:51:06,107 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:51:06,107 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21.store: b 2017-10-03 21:51:06,107 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:51:06,108 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21.store: c 2017-10-03 21:51:06,108 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:51:06,123 INFO [IPC Server handler 5 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741875_1051{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 4668 2017-10-03 21:51:06,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=60 2017-10-03 21:51:06,524 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=6, memsize=59, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/.tmp/a/b2cd70011fc04c5b9994465ea3b741d8 2017-10-03 21:51:06,545 INFO [IPC Server handler 8 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741876_1052{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|FINALIZED]]} size 0 2017-10-03 21:51:06,546 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=6, memsize=59, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/.tmp/b/d618e5ed140f478d8248c41d4fb5e7ad 2017-10-03 21:51:06,563 INFO [IPC Server handler 3 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741877_1053{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 0 2017-10-03 21:51:06,564 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=6, memsize=59, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/.tmp/c/22f802c9b4a8488088bf157b2379860b 2017-10-03 21:51:06,572 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/.tmp/a/b2cd70011fc04c5b9994465ea3b741d8 as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/a/b2cd70011fc04c5b9994465ea3b741d8 2017-10-03 21:51:06,580 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/a/b2cd70011fc04c5b9994465ea3b741d8, entries=1, sequenceid=6, filesize=4.6 K 2017-10-03 21:51:06,581 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/.tmp/b/d618e5ed140f478d8248c41d4fb5e7ad as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/b/d618e5ed140f478d8248c41d4fb5e7ad 2017-10-03 21:51:06,588 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/b/d618e5ed140f478d8248c41d4fb5e7ad, entries=1, sequenceid=6, filesize=4.6 K 2017-10-03 21:51:06,589 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/.tmp/c/22f802c9b4a8488088bf157b2379860b as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/c/22f802c9b4a8488088bf157b2379860b 2017-10-03 21:51:06,595 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/c/22f802c9b4a8488088bf157b2379860b, entries=1, sequenceid=6, filesize=4.6 K 2017-10-03 21:51:06,597 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(2731): Finished memstore flush of ~177 B/177, currentsize=0 B/0 for region TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. in 491ms, sequenceid=6, compaction requested=false 2017-10-03 21:51:06,598 INFO [StoreCloserThread-TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21.-1] regionserver.HStore(926): Closed a 2017-10-03 21:51:06,599 INFO [StoreCloserThread-TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21.-1] regionserver.HStore(926): Closed b 2017-10-03 21:51:06,600 INFO [StoreCloserThread-TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21.-1] regionserver.HStore(926): Closed c 2017-10-03 21:51:06,606 DEBUG [RS_CLOSE_REGION-cn012:37573-0] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/recovered.edits/9.seqid, newSeqId=9, maxSeqId=2 2017-10-03 21:51:06,607 DEBUG [RS_CLOSE_REGION-cn012:37573-0] coprocessor.CoprocessorHost(291): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2017-10-03 21:51:06,607 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(1712): Closed TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. 2017-10-03 21:51:06,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report CLOSED seqId=-1, pid=61, ppid=60, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testRegionObserver, region=e7270d04ea8b4e52e1e1ccdbe7082c21, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:06,609 DEBUG [RS_CLOSE_REGION-cn012:37573-0] handler.CloseRegionHandler(123): Closed TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. 2017-10-03 21:51:06,609 INFO [ProcExecWrkr-16] assignment.RegionStateStore(186): pid=61 updating hbase:meta row=TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21., regionState=CLOSED 2017-10-03 21:51:06,680 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=61, ppid=60, state=SUCCESS; UnassignProcedure table=TestTable.testRegionObserver, region=e7270d04ea8b4e52e1e1ccdbe7082c21, server=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:06,755 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=60, state=RUNNABLE:DISABLE_TABLE_SET_DISABLED_TABLE_STATE; DisableTableProcedure table=TestTable.testRegionObserver; resume parent processing. 2017-10-03 21:51:06,756 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1246): Finished pid=61, ppid=60, state=SUCCESS; UnassignProcedure table=TestTable.testRegionObserver, region=e7270d04ea8b4e52e1e1ccdbe7082c21, server=cn012.l42scl.hortonworks.com,37573,1507067090360 in 712msec 2017-10-03 21:51:06,756 DEBUG [ProcExecWrkr-20] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testRegionObserver","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067466755}]}} 2017-10-03 21:51:06,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=60 2017-10-03 21:51:06,760 INFO [ProcExecWrkr-20] hbase.MetaTableAccessor(1770): Updated table TestTable.testRegionObserver state to DISABLED in META 2017-10-03 21:51:06,761 INFO [ProcExecWrkr-20] procedure.DisableTableProcedure(285): Disabled table, TestTable.testRegionObserver, is completed. 2017-10-03 21:51:06,889 INFO [ProcExecWrkr-20] procedure2.ProcedureExecutor(1246): Finished pid=60, state=SUCCESS; DisableTableProcedure table=TestTable.testRegionObserver in 1.3370sec 2017-10-03 21:51:06,977 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 173091ms 2017-10-03 21:51:06,977 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 33473ms 2017-10-03 21:51:07,301 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2017-10-03 21:51:07,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=60 2017-10-03 21:51:07,760 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DISABLE, Table Name: default:TestTable.testRegionObserver completed 2017-10-03 21:51:07,762 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$5(2109): Client=hbase//172.18.128.12 delete TestTable.testRegionObserver 2017-10-03 21:51:07,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=62, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestTable.testRegionObserver 2017-10-03 21:51:07,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=62 2017-10-03 21:51:07,957 DEBUG [ProcExecWrkr-18] procedure.DeleteTableProcedure(99): Waiting for 'TestTable.testRegionObserver' regions in transition 2017-10-03 21:51:07,957 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 98261ms 2017-10-03 21:51:07,957 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 277225ms 2017-10-03 21:51:08,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=62 2017-10-03 21:51:08,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=62 2017-10-03 21:51:08,316 WARN [DataXceiver for client DFSClient_NONMAPREDUCE_-174551704_1 at /127.0.0.1:58812 [Receiving block BP-608549805-172.18.128.12-1507067084534:blk_1073741829_1005]] datanode.BlockReceiver(391): Slow flushOrSync took 358ms (threshold=300ms), isSync:true, flushTotalNanos=7457ns 2017-10-03 21:51:08,317 DEBUG [ProcExecWrkr-18] procedure.DeleteTableProcedure(110): delete 'TestTable.testRegionObserver' regions from META 2017-10-03 21:51:08,317 DEBUG [ProcExecWrkr-18] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067468317}]},"ts":9223372036854775807} 2017-10-03 21:51:08,321 INFO [ProcExecWrkr-18] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:51:08,322 DEBUG [ProcExecWrkr-18] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => e7270d04ea8b4e52e1e1ccdbe7082c21, NAME => 'TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:51:08,406 DEBUG [ProcExecWrkr-18] procedure.DeleteTableProcedure(115): delete 'TestTable.testRegionObserver' from filesystem 2017-10-03 21:51:08,413 DEBUG [ProcExecWrkr-18] procedure.DeleteTableProcedure(315): Archiving region TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21. from FS 2017-10-03 21:51:08,413 DEBUG [ProcExecWrkr-18] backup.HFileArchiver(114): ARCHIVING hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21 2017-10-03 21:51:08,419 DEBUG [ProcExecWrkr-18] backup.HFileArchiver(155): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/recovered.edits] 2017-10-03 21:51:08,428 DEBUG [ProcExecWrkr-18] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/a/b2cd70011fc04c5b9994465ea3b741d8, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/a/b2cd70011fc04c5b9994465ea3b741d8 2017-10-03 21:51:08,435 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:51:08,436 DEBUG [ProcExecWrkr-18] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/b/d618e5ed140f478d8248c41d4fb5e7ad, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/b/d618e5ed140f478d8248c41d4fb5e7ad 2017-10-03 21:51:08,442 DEBUG [ProcExecWrkr-18] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/c/22f802c9b4a8488088bf157b2379860b, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/c/22f802c9b4a8488088bf157b2379860b 2017-10-03 21:51:08,447 DEBUG [ProcExecWrkr-18] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/recovered.edits/9.seqid, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21/recovered.edits/9.seqid 2017-10-03 21:51:08,448 INFO [IPC Server handler 5 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741874_1050 127.0.0.1:41361 2017-10-03 21:51:08,448 DEBUG [ProcExecWrkr-18] backup.HFileArchiver(466): Deleted hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testRegionObserver/e7270d04ea8b4e52e1e1ccdbe7082c21 2017-10-03 21:51:08,448 DEBUG [ProcExecWrkr-18] procedure.DeleteTableProcedure(319): Table 'TestTable.testRegionObserver' archived! 2017-10-03 21:51:08,450 INFO [IPC Server handler 7 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741873_1049 127.0.0.1:41361 2017-10-03 21:51:08,488 DEBUG [ProcExecWrkr-18] procedure.DeleteTableProcedure(121): delete 'TestTable.testRegionObserver' descriptor 2017-10-03 21:51:08,489 DEBUG [ProcExecWrkr-18] procedure.DeleteTableProcedure(399): Removing 'TestTable.testRegionObserver' descriptor. 2017-10-03 21:51:08,530 DEBUG [ProcExecWrkr-18] procedure.DeleteTableProcedure(126): delete 'TestTable.testRegionObserver' assignment state 2017-10-03 21:51:08,530 DEBUG [ProcExecWrkr-18] procedure.DeleteTableProcedure(389): Removing 'TestTable.testRegionObserver' from region states. 2017-10-03 21:51:08,531 DEBUG [ProcExecWrkr-18] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067468530}]},"ts":9223372036854775807} 2017-10-03 21:51:08,535 INFO [ProcExecWrkr-18] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:51:08,536 DEBUG [ProcExecWrkr-18] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => e7270d04ea8b4e52e1e1ccdbe7082c21, NAME => 'TestTable.testRegionObserver,,1507067462923.e7270d04ea8b4e52e1e1ccdbe7082c21.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:51:08,536 DEBUG [ProcExecWrkr-18] procedure.DeleteTableProcedure(393): Marking 'TestTable.testRegionObserver' as deleted. 2017-10-03 21:51:08,536 DEBUG [ProcExecWrkr-18] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"TestTable.testRegionObserver","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":1507067468536}]},"ts":9223372036854775807} 2017-10-03 21:51:08,539 INFO [ProcExecWrkr-18] hbase.MetaTableAccessor(1796): Deleted table TestTable.testRegionObserver state from META 2017-10-03 21:51:08,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=62 2017-10-03 21:51:08,580 DEBUG [ProcExecWrkr-18] procedure.DeleteTableProcedure(132): delete 'TestTable.testRegionObserver' completed 2017-10-03 21:51:08,638 INFO [ProcExecWrkr-18] procedure2.ProcedureExecutor(1246): Finished pid=62, state=SUCCESS; DeleteTableProcedure table=TestTable.testRegionObserver in 818msec 2017-10-03 21:51:08,921 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 164428ms 2017-10-03 21:51:08,921 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 162614ms 2017-10-03 21:51:09,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=62 2017-10-03 21:51:09,063 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DELETE, Table Name: default:TestTable.testRegionObserver completed 2017-10-03 21:51:09,095 INFO [main] hbase.ResourceChecker(172): after: coprocessor.TestRegionObserverInterface#testRegionObserver Thread=305 (was 300) - Thread LEAK? -, OpenFileDescriptor=1246 (was 1246), MaxFileDescriptor=32000 (was 32000), SystemLoadAverage=390 (was 398), ProcessCount=347 (was 349), AvailableMemoryMB=48826 (was 48583) - AvailableMemoryMB LEAK? - 2017-10-03 21:51:09,095 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1246 is superior to 1024 2017-10-03 21:51:09,127 INFO [main] hbase.ResourceChecker(148): before: coprocessor.TestRegionObserverInterface#testIncrementHook Thread=305, OpenFileDescriptor=1246, MaxFileDescriptor=32000, SystemLoadAverage=390, ProcessCount=347, AvailableMemoryMB=48824 2017-10-03 21:51:09,127 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1246 is superior to 1024 2017-10-03 21:51:09,132 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$3(1748): Client=hbase//172.18.128.12 create 'TestTable.testIncrementHook', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2017-10-03 21:51:09,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=63, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestTable.testIncrementHook 2017-10-03 21:51:09,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=63 2017-10-03 21:51:09,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=63 2017-10-03 21:51:09,432 INFO [IPC Server handler 5 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741878_1054{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 1442 2017-10-03 21:51:09,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=63 2017-10-03 21:51:09,835 DEBUG [ProcExecWrkr-9] util.FSTableDescriptors(728): Wrote descriptor into: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testIncrementHook/.tabledesc/.tableinfo.0000000001 2017-10-03 21:51:09,837 INFO [RegionOpenAndInitThread-TestTable.testIncrementHook-1] regionserver.HRegion(6477): creating HRegion TestTable.testIncrementHook HTD == 'TestTable.testIncrementHook', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp Table name == TestTable.testIncrementHook 2017-10-03 21:51:09,846 INFO [IPC Server handler 1 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741879_1055{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 0 2017-10-03 21:51:09,847 DEBUG [RegionOpenAndInitThread-TestTable.testIncrementHook-1] regionserver.HRegion(798): Instantiated TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. 2017-10-03 21:51:09,847 DEBUG [RegionOpenAndInitThread-TestTable.testIncrementHook-1] regionserver.HRegion(1560): Closing TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95.: disabling compactions & flushes 2017-10-03 21:51:09,847 DEBUG [RegionOpenAndInitThread-TestTable.testIncrementHook-1] regionserver.HRegion(1599): Updates disabled for region TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. 2017-10-03 21:51:09,848 INFO [RegionOpenAndInitThread-TestTable.testIncrementHook-1] regionserver.HRegion(1712): Closed TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. 2017-10-03 21:51:09,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=63 2017-10-03 21:51:09,916 DEBUG [ProcExecWrkr-9] hbase.MetaTableAccessor(1417): Put{"totalColumns":1,"row":"TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95.","families":{"info":[{"qualifier":"regioninfo","vlen":61,"tag":[],"timestamp":9223372036854775807}]}} 2017-10-03 21:51:09,922 INFO [ProcExecWrkr-9] hbase.MetaTableAccessor(1614): Added 1 2017-10-03 21:51:09,953 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 24882ms 2017-10-03 21:51:09,953 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 277132ms 2017-10-03 21:51:09,964 DEBUG [ProcExecWrkr-9] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testIncrementHook","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067469964}]}} 2017-10-03 21:51:09,969 INFO [ProcExecWrkr-9] hbase.MetaTableAccessor(1770): Updated table TestTable.testIncrementHook state to ENABLING in META 2017-10-03 21:51:09,970 INFO [ProcExecWrkr-9] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testIncrementHook, region=180634dfc337c7bdeaedf9b59bfe6c95}] 2017-10-03 21:51:10,025 INFO [ProcExecWrkr-19] procedure.MasterProcedureScheduler(814): pid=64, ppid=63, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testIncrementHook, region=180634dfc337c7bdeaedf9b59bfe6c95 TestTable.testIncrementHook TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. 2017-10-03 21:51:10,028 INFO [ProcExecWrkr-19] assignment.AssignProcedure(204): Start pid=64, ppid=63, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testIncrementHook, region=180634dfc337c7bdeaedf9b59bfe6c95; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2017-10-03 21:51:10,180 INFO [ProcExecWrkr-21] assignment.RegionStateStore(186): pid=64 updating hbase:meta row=TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95., regionState=OPENING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:10,195 INFO [ProcExecWrkr-21] assignment.RegionTransitionProcedure(213): Dispatch pid=64, ppid=63, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testIncrementHook, region=180634dfc337c7bdeaedf9b59bfe6c95; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:10,346 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:51:10,349 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1916): Open TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. 2017-10-03 21:51:10,359 DEBUG [RS_OPEN_REGION-cn012:37573-0] regionserver.HRegion(6654): Opening region: {ENCODED => 180634dfc337c7bdeaedf9b59bfe6c95, NAME => 'TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95.', STARTKEY => '', ENDKEY => ''} 2017-10-03 21:51:10,360 INFO [RS_OPEN_REGION-cn012:37573-0] coprocessor.CoprocessorHost(164): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2017-10-03 21:51:10,360 DEBUG [RS_OPEN_REGION-cn012:37573-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table TestTable.testIncrementHook 180634dfc337c7bdeaedf9b59bfe6c95 2017-10-03 21:51:10,361 DEBUG [RS_OPEN_REGION-cn012:37573-0] regionserver.HRegion(798): Instantiated TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. 2017-10-03 21:51:10,368 DEBUG [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/a 2017-10-03 21:51:10,368 DEBUG [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/a 2017-10-03 21:51:10,368 INFO [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:51:10,368 INFO [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:51:10,369 INFO [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] hfile.CacheConfig(258): Created cacheConfig for a: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:51:10,369 INFO [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:51:10,373 DEBUG [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/b 2017-10-03 21:51:10,373 DEBUG [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/b 2017-10-03 21:51:10,374 INFO [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:51:10,374 INFO [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:51:10,374 INFO [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] hfile.CacheConfig(258): Created cacheConfig for b: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:51:10,375 INFO [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:51:10,379 DEBUG [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/c 2017-10-03 21:51:10,379 DEBUG [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/c 2017-10-03 21:51:10,379 INFO [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:51:10,380 INFO [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:51:10,380 INFO [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] hfile.CacheConfig(258): Created cacheConfig for c: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:51:10,380 INFO [StoreOpener-180634dfc337c7bdeaedf9b59bfe6c95-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:51:10,381 INFO [RS_OPEN_REGION-cn012:37573-0] regionserver.HRegion(1030): Setting FlushNonSloppyStoresFirstPolicy for the region=TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. 2017-10-03 21:51:10,382 DEBUG [RS_OPEN_REGION-cn012:37573-0] regionserver.HRegion(4121): Found 0 recovered edits file(s) under hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95 2017-10-03 21:51:10,385 DEBUG [RS_OPEN_REGION-cn012:37573-0] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table TestTable.testIncrementHook, use config (44739242) instead 2017-10-03 21:51:10,389 DEBUG [RS_OPEN_REGION-cn012:37573-0] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2017-10-03 21:51:10,389 INFO [RS_OPEN_REGION-cn012:37573-0] regionserver.HRegion(957): Onlined 180634dfc337c7bdeaedf9b59bfe6c95; next sequenceid=2 2017-10-03 21:51:10,391 INFO [PostOpenDeployTasks:180634dfc337c7bdeaedf9b59bfe6c95] regionserver.HRegionServer(2170): Post open deploy tasks for TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. 2017-10-03 21:51:10,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=64, ppid=63, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testIncrementHook, region=180634dfc337c7bdeaedf9b59bfe6c95; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:10,394 DEBUG [PostOpenDeployTasks:180634dfc337c7bdeaedf9b59bfe6c95] regionserver.HRegionServer(2197): Finished post open deploy task for TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. 2017-10-03 21:51:10,394 INFO [ProcExecWrkr-23] assignment.RegionStateStore(186): pid=64 updating hbase:meta row=TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95., regionState=OPEN, openSeqNum=2, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:10,395 DEBUG [RS_OPEN_REGION-cn012:37573-0] handler.OpenRegionHandler(127): Opened TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. on cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:10,405 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@70efb718] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:41361 to delete [blk_1073741873_1049, blk_1073741874_1050] 2017-10-03 21:51:10,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=63 2017-10-03 21:51:10,448 INFO [ProcExecWrkr-23] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=64, ppid=63, state=SUCCESS; AssignProcedure table=TestTable.testIncrementHook, region=180634dfc337c7bdeaedf9b59bfe6c95 2017-10-03 21:51:10,497 INFO [ProcExecWrkr-23] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=63, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=TestTable.testIncrementHook; resume parent processing. 2017-10-03 21:51:10,497 INFO [ProcExecWrkr-23] procedure2.ProcedureExecutor(1246): Finished pid=64, ppid=63, state=SUCCESS; AssignProcedure table=TestTable.testIncrementHook, region=180634dfc337c7bdeaedf9b59bfe6c95 in 429msec 2017-10-03 21:51:10,497 DEBUG [ProcExecWrkr-22] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testIncrementHook","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067470497}]}} 2017-10-03 21:51:10,502 INFO [ProcExecWrkr-22] hbase.MetaTableAccessor(1770): Updated table TestTable.testIncrementHook state to ENABLED in META 2017-10-03 21:51:10,702 INFO [ProcExecWrkr-22] procedure2.ProcedureExecutor(1246): Finished pid=63, state=SUCCESS; CreateTableProcedure table=TestTable.testIncrementHook in 1.4780sec 2017-10-03 21:51:10,875 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(750): java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808) at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2073) at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.getCurrentBlockReplication(HdfsDataOutputStream.java:78) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.getLogReplication(FSHLog.java:747) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLowReplication(FSHLog.java:664) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.checkLogRoll(FSHLog.java:646) at org.apache.hadoop.hbase.regionserver.LogRoller.checkLowReplication(LogRoller.java:130) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:143) at java.lang.Thread.run(Thread.java:748) 2017-10-03 21:51:10,920 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 81203ms 2017-10-03 21:51:10,920 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 195840ms 2017-10-03 21:51:11,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=63 2017-10-03 21:51:11,412 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: CREATE, Table Name: default:TestTable.testIncrementHook completed 2017-10-03 21:51:11,414 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:51:11,427 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:51:11,451 INFO [Time-limited test] client.HBaseAdmin$15(919): Started disable of TestTable.testIncrementHook 2017-10-03 21:51:11,451 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$11(2331): Client=hbase//172.18.128.12 disable TestTable.testIncrementHook 2017-10-03 21:51:11,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=65, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestTable.testIncrementHook 2017-10-03 21:51:11,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=65 2017-10-03 21:51:11,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=65 2017-10-03 21:51:11,822 DEBUG [ProcExecWrkr-1] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testIncrementHook","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067471821}]}} 2017-10-03 21:51:11,827 INFO [ProcExecWrkr-1] hbase.MetaTableAccessor(1770): Updated table TestTable.testIncrementHook state to DISABLING in META 2017-10-03 21:51:11,913 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=66, ppid=65, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testIncrementHook, region=180634dfc337c7bdeaedf9b59bfe6c95, server=cn012.l42scl.hortonworks.com,37573,1507067090360}] 2017-10-03 21:51:11,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=65 2017-10-03 21:51:11,945 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 297294ms 2017-10-03 21:51:11,945 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 12159ms 2017-10-03 21:51:11,989 INFO [ProcExecWrkr-1] procedure.MasterProcedureScheduler(814): pid=66, ppid=65, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testIncrementHook, region=180634dfc337c7bdeaedf9b59bfe6c95, server=cn012.l42scl.hortonworks.com,37573,1507067090360 TestTable.testIncrementHook TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. 2017-10-03 21:51:11,989 INFO [ProcExecWrkr-1] assignment.RegionStateStore(186): pid=66 updating hbase:meta row=TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95., regionState=CLOSING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:11,995 INFO [ProcExecWrkr-1] assignment.RegionTransitionProcedure(213): Dispatch pid=66, ppid=65, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testIncrementHook, region=180634dfc337c7bdeaedf9b59bfe6c95, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:12,145 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:51:12,147 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1520): Close 180634dfc337c7bdeaedf9b59bfe6c95 without moving 2017-10-03 21:51:12,147 DEBUG [RS_CLOSE_REGION-cn012:37573-1] handler.CloseRegionHandler(91): Processing close of TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. 2017-10-03 21:51:12,149 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(1560): Closing TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95.: disabling compactions & flushes 2017-10-03 21:51:12,149 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(1599): Updates disabled for region TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. 2017-10-03 21:51:12,149 INFO [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(2535): Flushing 3/3 column families, memstore=34 B 2017-10-03 21:51:12,149 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95.store: a 2017-10-03 21:51:12,149 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:51:12,149 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95.store: b 2017-10-03 21:51:12,150 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:0, and the number of cells in new segment is:0 2017-10-03 21:51:12,150 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95.store: c 2017-10-03 21:51:12,150 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:0, and the number of cells in new segment is:0 2017-10-03 21:51:12,162 INFO [IPC Server handler 5 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741880_1056{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 4667 2017-10-03 21:51:12,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=65 2017-10-03 21:51:12,563 INFO [RS_CLOSE_REGION-cn012:37573-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=5, memsize=34, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/.tmp/a/bf203fdf0adc45abac040a039f8d428c 2017-10-03 21:51:12,574 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/.tmp/a/bf203fdf0adc45abac040a039f8d428c as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/a/bf203fdf0adc45abac040a039f8d428c 2017-10-03 21:51:12,582 INFO [RS_CLOSE_REGION-cn012:37573-1] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/a/bf203fdf0adc45abac040a039f8d428c, entries=1, sequenceid=5, filesize=4.6 K 2017-10-03 21:51:12,584 INFO [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(2731): Finished memstore flush of ~34 B/34, currentsize=0 B/0 for region TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. in 435ms, sequenceid=5, compaction requested=false 2017-10-03 21:51:12,586 INFO [StoreCloserThread-TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95.-1] regionserver.HStore(926): Closed a 2017-10-03 21:51:12,586 INFO [StoreCloserThread-TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95.-1] regionserver.HStore(926): Closed b 2017-10-03 21:51:12,587 INFO [StoreCloserThread-TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95.-1] regionserver.HStore(926): Closed c 2017-10-03 21:51:12,592 DEBUG [RS_CLOSE_REGION-cn012:37573-1] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/recovered.edits/8.seqid, newSeqId=8, maxSeqId=2 2017-10-03 21:51:12,592 DEBUG [RS_CLOSE_REGION-cn012:37573-1] coprocessor.CoprocessorHost(291): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2017-10-03 21:51:12,593 INFO [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(1712): Closed TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. 2017-10-03 21:51:12,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report CLOSED seqId=-1, pid=66, ppid=65, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testIncrementHook, region=180634dfc337c7bdeaedf9b59bfe6c95, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:12,594 DEBUG [RS_CLOSE_REGION-cn012:37573-1] handler.CloseRegionHandler(123): Closed TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. 2017-10-03 21:51:12,594 INFO [ProcExecWrkr-2] assignment.RegionStateStore(186): pid=66 updating hbase:meta row=TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95., regionState=CLOSED 2017-10-03 21:51:12,669 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=66, ppid=65, state=SUCCESS; UnassignProcedure table=TestTable.testIncrementHook, region=180634dfc337c7bdeaedf9b59bfe6c95, server=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:12,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=65 2017-10-03 21:51:12,791 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=65, state=RUNNABLE:DISABLE_TABLE_SET_DISABLED_TABLE_STATE; DisableTableProcedure table=TestTable.testIncrementHook; resume parent processing. 2017-10-03 21:51:12,792 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1246): Finished pid=66, ppid=65, state=SUCCESS; UnassignProcedure table=TestTable.testIncrementHook, region=180634dfc337c7bdeaedf9b59bfe6c95, server=cn012.l42scl.hortonworks.com,37573,1507067090360 in 685msec 2017-10-03 21:51:12,792 DEBUG [ProcExecWrkr-6] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testIncrementHook","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067472791}]}} 2017-10-03 21:51:12,796 INFO [ProcExecWrkr-6] hbase.MetaTableAccessor(1770): Updated table TestTable.testIncrementHook state to DISABLED in META 2017-10-03 21:51:12,796 INFO [ProcExecWrkr-6] procedure.DisableTableProcedure(285): Disabled table, TestTable.testIncrementHook, is completed. 2017-10-03 21:51:12,921 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 291253ms 2017-10-03 21:51:12,921 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 86745ms 2017-10-03 21:51:12,964 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1246): Finished pid=65, state=SUCCESS; DisableTableProcedure table=TestTable.testIncrementHook in 1.4560sec 2017-10-03 21:51:13,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=65 2017-10-03 21:51:13,735 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DISABLE, Table Name: default:TestTable.testIncrementHook completed 2017-10-03 21:51:13,737 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$5(2109): Client=hbase//172.18.128.12 delete TestTable.testIncrementHook 2017-10-03 21:51:13,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=67, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestTable.testIncrementHook 2017-10-03 21:51:13,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=67 2017-10-03 21:51:13,892 DEBUG [ProcExecWrkr-4] procedure.DeleteTableProcedure(99): Waiting for 'TestTable.testIncrementHook' regions in transition 2017-10-03 21:51:13,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 37436ms 2017-10-03 21:51:13,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 134870ms 2017-10-03 21:51:13,954 DEBUG [ProcExecWrkr-4] procedure.DeleteTableProcedure(110): delete 'TestTable.testIncrementHook' regions from META 2017-10-03 21:51:13,955 DEBUG [ProcExecWrkr-4] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067473954}]},"ts":9223372036854775807} 2017-10-03 21:51:13,962 INFO [ProcExecWrkr-4] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:51:13,962 DEBUG [ProcExecWrkr-4] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => 180634dfc337c7bdeaedf9b59bfe6c95, NAME => 'TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:51:13,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=67 2017-10-03 21:51:14,038 DEBUG [ProcExecWrkr-4] procedure.DeleteTableProcedure(115): delete 'TestTable.testIncrementHook' from filesystem 2017-10-03 21:51:14,045 DEBUG [ProcExecWrkr-4] procedure.DeleteTableProcedure(315): Archiving region TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95. from FS 2017-10-03 21:51:14,046 DEBUG [ProcExecWrkr-4] backup.HFileArchiver(114): ARCHIVING hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95 2017-10-03 21:51:14,054 DEBUG [ProcExecWrkr-4] backup.HFileArchiver(155): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/recovered.edits] 2017-10-03 21:51:14,066 DEBUG [ProcExecWrkr-4] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/a/bf203fdf0adc45abac040a039f8d428c, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/a/bf203fdf0adc45abac040a039f8d428c 2017-10-03 21:51:14,081 DEBUG [ProcExecWrkr-4] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/recovered.edits/8.seqid, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95/recovered.edits/8.seqid 2017-10-03 21:51:14,082 INFO [IPC Server handler 8 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741879_1055 127.0.0.1:41361 2017-10-03 21:51:14,083 DEBUG [ProcExecWrkr-4] backup.HFileArchiver(466): Deleted hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testIncrementHook/180634dfc337c7bdeaedf9b59bfe6c95 2017-10-03 21:51:14,083 DEBUG [ProcExecWrkr-4] procedure.DeleteTableProcedure(319): Table 'TestTable.testIncrementHook' archived! 2017-10-03 21:51:14,085 INFO [IPC Server handler 2 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741878_1054 127.0.0.1:41361 2017-10-03 21:51:14,122 DEBUG [ProcExecWrkr-4] procedure.DeleteTableProcedure(121): delete 'TestTable.testIncrementHook' descriptor 2017-10-03 21:51:14,122 DEBUG [ProcExecWrkr-4] procedure.DeleteTableProcedure(399): Removing 'TestTable.testIncrementHook' descriptor. 2017-10-03 21:51:14,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=67 2017-10-03 21:51:14,198 DEBUG [ProcExecWrkr-4] procedure.DeleteTableProcedure(126): delete 'TestTable.testIncrementHook' assignment state 2017-10-03 21:51:14,198 DEBUG [ProcExecWrkr-4] procedure.DeleteTableProcedure(389): Removing 'TestTable.testIncrementHook' from region states. 2017-10-03 21:51:14,199 DEBUG [ProcExecWrkr-4] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067474199}]},"ts":9223372036854775807} 2017-10-03 21:51:14,206 INFO [ProcExecWrkr-4] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:51:14,206 DEBUG [ProcExecWrkr-4] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => 180634dfc337c7bdeaedf9b59bfe6c95, NAME => 'TestTable.testIncrementHook,,1507067469132.180634dfc337c7bdeaedf9b59bfe6c95.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:51:14,206 DEBUG [ProcExecWrkr-4] procedure.DeleteTableProcedure(393): Marking 'TestTable.testIncrementHook' as deleted. 2017-10-03 21:51:14,206 DEBUG [ProcExecWrkr-4] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"TestTable.testIncrementHook","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":1507067474206}]},"ts":9223372036854775807} 2017-10-03 21:51:14,210 INFO [ProcExecWrkr-4] hbase.MetaTableAccessor(1796): Deleted table TestTable.testIncrementHook state from META 2017-10-03 21:51:14,247 DEBUG [ProcExecWrkr-4] procedure.DeleteTableProcedure(132): delete 'TestTable.testIncrementHook' completed 2017-10-03 21:51:14,289 INFO [ProcExecWrkr-4] procedure2.ProcedureExecutor(1246): Finished pid=67, state=SUCCESS; DeleteTableProcedure table=TestTable.testIncrementHook in 510msec 2017-10-03 21:51:14,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=67 2017-10-03 21:51:14,497 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DELETE, Table Name: default:TestTable.testIncrementHook completed 2017-10-03 21:51:14,538 INFO [main] hbase.ResourceChecker(172): after: coprocessor.TestRegionObserverInterface#testIncrementHook Thread=312 (was 305) - Thread LEAK? -, OpenFileDescriptor=1246 (was 1246), MaxFileDescriptor=32000 (was 32000), SystemLoadAverage=367 (was 390), ProcessCount=347 (was 347), AvailableMemoryMB=48673 (was 48824) 2017-10-03 21:51:14,539 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1246 is superior to 1024 2017-10-03 21:51:14,575 INFO [main] hbase.ResourceChecker(148): before: coprocessor.TestRegionObserverInterface#testHBASE14489 Thread=312, OpenFileDescriptor=1246, MaxFileDescriptor=32000, SystemLoadAverage=367, ProcessCount=347, AvailableMemoryMB=48673 2017-10-03 21:51:14,576 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1246 is superior to 1024 2017-10-03 21:51:14,581 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$3(1748): Client=hbase//172.18.128.12 create 'testHBASE14489', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2017-10-03 21:51:14,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=68, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testHBASE14489 2017-10-03 21:51:14,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=68 2017-10-03 21:51:14,859 INFO [IPC Server handler 8 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741881_1057{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|RBW]]} size 509 2017-10-03 21:51:14,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=68 2017-10-03 21:51:14,997 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 53543ms 2017-10-03 21:51:14,997 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 97527ms 2017-10-03 21:51:15,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=68 2017-10-03 21:51:15,264 DEBUG [ProcExecWrkr-5] util.FSTableDescriptors(728): Wrote descriptor into: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBASE14489/.tabledesc/.tableinfo.0000000001 2017-10-03 21:51:15,266 INFO [RegionOpenAndInitThread-testHBASE14489-1] regionserver.HRegion(6477): creating HRegion testHBASE14489 HTD == 'testHBASE14489', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp Table name == testHBASE14489 2017-10-03 21:51:15,285 INFO [IPC Server handler 3 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741882_1058{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 49 2017-10-03 21:51:15,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=68 2017-10-03 21:51:15,687 DEBUG [RegionOpenAndInitThread-testHBASE14489-1] regionserver.HRegion(798): Instantiated testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. 2017-10-03 21:51:15,688 DEBUG [RegionOpenAndInitThread-testHBASE14489-1] regionserver.HRegion(1560): Closing testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d.: disabling compactions & flushes 2017-10-03 21:51:15,688 DEBUG [RegionOpenAndInitThread-testHBASE14489-1] regionserver.HRegion(1599): Updates disabled for region testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. 2017-10-03 21:51:15,688 INFO [RegionOpenAndInitThread-testHBASE14489-1] regionserver.HRegion(1712): Closed testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. 2017-10-03 21:51:15,798 DEBUG [ProcExecWrkr-5] hbase.MetaTableAccessor(1417): Put{"totalColumns":1,"row":"testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d.","families":{"info":[{"qualifier":"regioninfo","vlen":48,"tag":[],"timestamp":9223372036854775807}]}} 2017-10-03 21:51:15,804 INFO [ProcExecWrkr-5] hbase.MetaTableAccessor(1614): Added 1 2017-10-03 21:51:15,864 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2017-10-03 21:51:15,872 DEBUG [ProcExecWrkr-5] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"testHBASE14489","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067475872}]}} 2017-10-03 21:51:15,877 INFO [ProcExecWrkr-5] hbase.MetaTableAccessor(1770): Updated table testHBASE14489 state to ENABLING in META 2017-10-03 21:51:15,877 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=69, ppid=68, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testHBASE14489, region=cc10fbe370733c636d80e9876212665d}] 2017-10-03 21:51:15,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=68 2017-10-03 21:51:15,917 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 107367ms 2017-10-03 21:51:15,917 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 79306ms 2017-10-03 21:51:15,947 INFO [ProcExecWrkr-3] procedure.MasterProcedureScheduler(814): pid=69, ppid=68, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testHBASE14489, region=cc10fbe370733c636d80e9876212665d testHBASE14489 testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. 2017-10-03 21:51:15,951 INFO [ProcExecWrkr-3] assignment.AssignProcedure(204): Start pid=69, ppid=68, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testHBASE14489, region=cc10fbe370733c636d80e9876212665d; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2017-10-03 21:51:16,103 INFO [ProcExecWrkr-7] assignment.RegionStateStore(186): pid=69 updating hbase:meta row=testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d., regionState=OPENING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:16,116 INFO [ProcExecWrkr-7] assignment.RegionTransitionProcedure(213): Dispatch pid=69, ppid=68, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testHBASE14489, region=cc10fbe370733c636d80e9876212665d; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:16,266 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:51:16,267 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1916): Open testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. 2017-10-03 21:51:16,272 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(6654): Opening region: {ENCODED => cc10fbe370733c636d80e9876212665d, NAME => 'testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d.', STARTKEY => '', ENDKEY => ''} 2017-10-03 21:51:16,273 INFO [RS_OPEN_REGION-cn012:37573-1] coprocessor.CoprocessorHost(164): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2017-10-03 21:51:16,273 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testHBASE14489 cc10fbe370733c636d80e9876212665d 2017-10-03 21:51:16,273 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(798): Instantiated testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. 2017-10-03 21:51:16,277 DEBUG [StoreOpener-cc10fbe370733c636d80e9876212665d-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d/a 2017-10-03 21:51:16,277 DEBUG [StoreOpener-cc10fbe370733c636d80e9876212665d-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d/a 2017-10-03 21:51:16,278 INFO [StoreOpener-cc10fbe370733c636d80e9876212665d-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 33554432 2017-10-03 21:51:16,278 INFO [StoreOpener-cc10fbe370733c636d80e9876212665d-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:51:16,278 INFO [StoreOpener-cc10fbe370733c636d80e9876212665d-1] hfile.CacheConfig(258): Created cacheConfig for a: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:51:16,278 INFO [StoreOpener-cc10fbe370733c636d80e9876212665d-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:51:16,279 INFO [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(1030): Setting FlushNonSloppyStoresFirstPolicy for the region=testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. 2017-10-03 21:51:16,280 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(4121): Found 0 recovered edits file(s) under hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d 2017-10-03 21:51:16,282 DEBUG [RS_OPEN_REGION-cn012:37573-1] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table testHBASE14489, use config (134217728) instead 2017-10-03 21:51:16,285 DEBUG [RS_OPEN_REGION-cn012:37573-1] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2017-10-03 21:51:16,285 INFO [RS_OPEN_REGION-cn012:37573-1] regionserver.HRegion(957): Onlined cc10fbe370733c636d80e9876212665d; next sequenceid=2 2017-10-03 21:51:16,287 INFO [PostOpenDeployTasks:cc10fbe370733c636d80e9876212665d] regionserver.HRegionServer(2170): Post open deploy tasks for testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. 2017-10-03 21:51:16,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=69, ppid=68, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testHBASE14489, region=cc10fbe370733c636d80e9876212665d; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:16,289 INFO [ProcExecWrkr-10] assignment.RegionStateStore(186): pid=69 updating hbase:meta row=testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d., regionState=OPEN, openSeqNum=2, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:16,289 DEBUG [PostOpenDeployTasks:cc10fbe370733c636d80e9876212665d] regionserver.HRegionServer(2197): Finished post open deploy task for testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. 2017-10-03 21:51:16,290 DEBUG [RS_OPEN_REGION-cn012:37573-1] handler.OpenRegionHandler(127): Opened testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. on cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:16,386 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2017-10-03 21:51:16,398 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=69, ppid=68, state=SUCCESS; AssignProcedure table=testHBASE14489, region=cc10fbe370733c636d80e9876212665d 2017-10-03 21:51:16,405 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@70efb718] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:41361 to delete [blk_1073741878_1054, blk_1073741879_1055] 2017-10-03 21:51:16,499 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=68, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testHBASE14489; resume parent processing. 2017-10-03 21:51:16,499 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1246): Finished pid=69, ppid=68, state=SUCCESS; AssignProcedure table=testHBASE14489, region=cc10fbe370733c636d80e9876212665d in 416msec 2017-10-03 21:51:16,500 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"testHBASE14489","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067476499}]}} 2017-10-03 21:51:16,505 INFO [ProcExecWrkr-11] hbase.MetaTableAccessor(1770): Updated table testHBASE14489 state to ENABLED in META 2017-10-03 21:51:16,734 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1246): Finished pid=68, state=SUCCESS; CreateTableProcedure table=testHBASE14489 in 2.0220sec 2017-10-03 21:51:16,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=68 2017-10-03 21:51:16,890 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: CREATE, Table Name: default:testHBASE14489 completed 2017-10-03 21:51:16,891 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:51:16,900 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:51:16,918 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 283104ms 2017-10-03 21:51:16,918 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 113677ms 2017-10-03 21:51:16,920 INFO [Time-limited test] client.HBaseAdmin$15(919): Started disable of testHBASE14489 2017-10-03 21:51:16,921 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$11(2331): Client=hbase//172.18.128.12 disable testHBASE14489 2017-10-03 21:51:17,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=70, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testHBASE14489 2017-10-03 21:51:17,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=70 2017-10-03 21:51:17,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=70 2017-10-03 21:51:17,265 DEBUG [ProcExecWrkr-8] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"testHBASE14489","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067477264}]}} 2017-10-03 21:51:17,270 INFO [ProcExecWrkr-8] hbase.MetaTableAccessor(1770): Updated table testHBASE14489 state to DISABLING in META 2017-10-03 21:51:17,361 INFO [ProcExecWrkr-8] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=71, ppid=70, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testHBASE14489, region=cc10fbe370733c636d80e9876212665d, server=cn012.l42scl.hortonworks.com,37573,1507067090360}] 2017-10-03 21:51:17,397 INFO [ProcExecWrkr-8] procedure.MasterProcedureScheduler(814): pid=71, ppid=70, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testHBASE14489, region=cc10fbe370733c636d80e9876212665d, server=cn012.l42scl.hortonworks.com,37573,1507067090360 testHBASE14489 testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. 2017-10-03 21:51:17,397 INFO [ProcExecWrkr-8] assignment.RegionStateStore(186): pid=71 updating hbase:meta row=testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d., regionState=CLOSING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:17,401 INFO [ProcExecWrkr-8] assignment.RegionTransitionProcedure(213): Dispatch pid=71, ppid=70, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testHBASE14489, region=cc10fbe370733c636d80e9876212665d, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:17,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=70 2017-10-03 21:51:17,552 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:51:17,554 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1520): Close cc10fbe370733c636d80e9876212665d without moving 2017-10-03 21:51:17,554 DEBUG [RS_CLOSE_REGION-cn012:37573-2] handler.CloseRegionHandler(91): Processing close of testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. 2017-10-03 21:51:17,556 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(1560): Closing testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d.: disabling compactions & flushes 2017-10-03 21:51:17,557 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(1599): Updates disabled for region testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. 2017-10-03 21:51:17,557 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(2535): Flushing 1/1 column families, memstore=30 B 2017-10-03 21:51:17,557 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d.store: a 2017-10-03 21:51:17,558 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:51:17,584 INFO [IPC Server handler 4 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741883_1059{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 4669 2017-10-03 21:51:17,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=70 2017-10-03 21:51:17,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 256620ms 2017-10-03 21:51:17,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 90929ms 2017-10-03 21:51:17,987 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=5, memsize=30, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d/.tmp/a/fdfc11f8cf4e4c338d0a10f52f8b4a9a 2017-10-03 21:51:18,001 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d/.tmp/a/fdfc11f8cf4e4c338d0a10f52f8b4a9a as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d/a/fdfc11f8cf4e4c338d0a10f52f8b4a9a 2017-10-03 21:51:18,012 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d/a/fdfc11f8cf4e4c338d0a10f52f8b4a9a, entries=1, sequenceid=5, filesize=4.6 K 2017-10-03 21:51:18,015 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(2731): Finished memstore flush of ~30 B/30, currentsize=0 B/0 for region testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. in 458ms, sequenceid=5, compaction requested=false 2017-10-03 21:51:18,019 INFO [StoreCloserThread-testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d.-1] regionserver.HStore(926): Closed a 2017-10-03 21:51:18,027 DEBUG [RS_CLOSE_REGION-cn012:37573-2] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d/recovered.edits/8.seqid, newSeqId=8, maxSeqId=2 2017-10-03 21:51:18,028 DEBUG [RS_CLOSE_REGION-cn012:37573-2] coprocessor.CoprocessorHost(291): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2017-10-03 21:51:18,029 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(1712): Closed testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. 2017-10-03 21:51:18,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report CLOSED seqId=-1, pid=71, ppid=70, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testHBASE14489, region=cc10fbe370733c636d80e9876212665d, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:18,032 DEBUG [RS_CLOSE_REGION-cn012:37573-2] handler.CloseRegionHandler(123): Closed testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. 2017-10-03 21:51:18,032 INFO [ProcExecWrkr-15] assignment.RegionStateStore(186): pid=71 updating hbase:meta row=testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d., regionState=CLOSED 2017-10-03 21:51:18,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=70 2017-10-03 21:51:18,308 INFO [ProcExecWrkr-15] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=71, ppid=70, state=SUCCESS; UnassignProcedure table=testHBASE14489, region=cc10fbe370733c636d80e9876212665d, server=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:18,400 INFO [ProcExecWrkr-15] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=70, state=RUNNABLE:DISABLE_TABLE_SET_DISABLED_TABLE_STATE; DisableTableProcedure table=testHBASE14489; resume parent processing. 2017-10-03 21:51:18,400 INFO [ProcExecWrkr-15] procedure2.ProcedureExecutor(1246): Finished pid=71, ppid=70, state=SUCCESS; UnassignProcedure table=testHBASE14489, region=cc10fbe370733c636d80e9876212665d, server=cn012.l42scl.hortonworks.com,37573,1507067090360 in 676msec 2017-10-03 21:51:18,400 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"testHBASE14489","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067478400}]}} 2017-10-03 21:51:18,406 INFO [ProcExecWrkr-14] hbase.MetaTableAccessor(1770): Updated table testHBASE14489 state to DISABLED in META 2017-10-03 21:51:18,406 INFO [ProcExecWrkr-14] procedure.DisableTableProcedure(285): Disabled table, testHBASE14489, is completed. 2017-10-03 21:51:18,575 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1246): Finished pid=70, state=SUCCESS; DisableTableProcedure table=testHBASE14489 in 1.5710sec 2017-10-03 21:51:18,918 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 53636ms 2017-10-03 21:51:18,919 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 200561ms 2017-10-03 21:51:19,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=70 2017-10-03 21:51:19,224 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DISABLE, Table Name: default:testHBASE14489 completed 2017-10-03 21:51:19,226 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$5(2109): Client=hbase//172.18.128.12 delete testHBASE14489 2017-10-03 21:51:19,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=72, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testHBASE14489 2017-10-03 21:51:19,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=72 2017-10-03 21:51:19,413 DEBUG [ProcExecWrkr-17] procedure.DeleteTableProcedure(99): Waiting for 'testHBASE14489' regions in transition 2017-10-03 21:51:19,487 DEBUG [ProcExecWrkr-17] procedure.DeleteTableProcedure(110): delete 'testHBASE14489' regions from META 2017-10-03 21:51:19,487 DEBUG [ProcExecWrkr-17] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067479487}]},"ts":9223372036854775807} 2017-10-03 21:51:19,493 INFO [ProcExecWrkr-17] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:51:19,493 DEBUG [ProcExecWrkr-17] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => cc10fbe370733c636d80e9876212665d, NAME => 'testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:51:19,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=72 2017-10-03 21:51:19,562 DEBUG [ProcExecWrkr-17] procedure.DeleteTableProcedure(115): delete 'testHBASE14489' from filesystem 2017-10-03 21:51:19,568 DEBUG [ProcExecWrkr-17] procedure.DeleteTableProcedure(315): Archiving region testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d. from FS 2017-10-03 21:51:19,568 DEBUG [ProcExecWrkr-17] backup.HFileArchiver(114): ARCHIVING hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d 2017-10-03 21:51:19,574 DEBUG [ProcExecWrkr-17] backup.HFileArchiver(155): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d/recovered.edits] 2017-10-03 21:51:19,585 DEBUG [ProcExecWrkr-17] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d/a/fdfc11f8cf4e4c338d0a10f52f8b4a9a, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d/a/fdfc11f8cf4e4c338d0a10f52f8b4a9a 2017-10-03 21:51:19,591 DEBUG [ProcExecWrkr-17] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d/recovered.edits/8.seqid, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d/recovered.edits/8.seqid 2017-10-03 21:51:19,592 INFO [IPC Server handler 7 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741882_1058 127.0.0.1:41361 2017-10-03 21:51:19,593 DEBUG [ProcExecWrkr-17] backup.HFileArchiver(466): Deleted hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/testHBASE14489/cc10fbe370733c636d80e9876212665d 2017-10-03 21:51:19,593 DEBUG [ProcExecWrkr-17] procedure.DeleteTableProcedure(319): Table 'testHBASE14489' archived! 2017-10-03 21:51:19,595 INFO [IPC Server handler 6 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741881_1057 127.0.0.1:41361 2017-10-03 21:51:19,681 DEBUG [ProcExecWrkr-17] procedure.DeleteTableProcedure(121): delete 'testHBASE14489' descriptor 2017-10-03 21:51:19,681 DEBUG [ProcExecWrkr-17] procedure.DeleteTableProcedure(399): Removing 'testHBASE14489' descriptor. 2017-10-03 21:51:19,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=72 2017-10-03 21:51:19,765 DEBUG [ProcExecWrkr-17] procedure.DeleteTableProcedure(126): delete 'testHBASE14489' assignment state 2017-10-03 21:51:19,765 DEBUG [ProcExecWrkr-17] procedure.DeleteTableProcedure(389): Removing 'testHBASE14489' from region states. 2017-10-03 21:51:19,766 DEBUG [ProcExecWrkr-17] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067479765}]},"ts":9223372036854775807} 2017-10-03 21:51:19,771 INFO [ProcExecWrkr-17] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:51:19,771 DEBUG [ProcExecWrkr-17] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => cc10fbe370733c636d80e9876212665d, NAME => 'testHBASE14489,,1507067474580.cc10fbe370733c636d80e9876212665d.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:51:19,771 DEBUG [ProcExecWrkr-17] procedure.DeleteTableProcedure(393): Marking 'testHBASE14489' as deleted. 2017-10-03 21:51:19,771 DEBUG [ProcExecWrkr-17] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"testHBASE14489","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":1507067479771}]},"ts":9223372036854775807} 2017-10-03 21:51:19,775 INFO [ProcExecWrkr-17] hbase.MetaTableAccessor(1796): Deleted table testHBASE14489 state from META 2017-10-03 21:51:19,848 DEBUG [ProcExecWrkr-17] procedure.DeleteTableProcedure(132): delete 'testHBASE14489' completed 2017-10-03 21:51:19,995 INFO [ProcExecWrkr-17] procedure2.ProcedureExecutor(1246): Finished pid=72, state=SUCCESS; DeleteTableProcedure table=testHBASE14489 in 623msec 2017-10-03 21:51:20,006 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 112312ms 2017-10-03 21:51:20,006 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 22320ms 2017-10-03 21:51:20,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=72 2017-10-03 21:51:20,017 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DELETE, Table Name: default:testHBASE14489 completed 2017-10-03 21:51:20,058 INFO [main] hbase.ResourceChecker(172): after: coprocessor.TestRegionObserverInterface#testHBASE14489 Thread=318 (was 312) - Thread LEAK? -, OpenFileDescriptor=1246 (was 1246), MaxFileDescriptor=32000 (was 32000), SystemLoadAverage=361 (was 367), ProcessCount=348 (was 347) - ProcessCount LEAK? -, AvailableMemoryMB=48672 (was 48673) 2017-10-03 21:51:20,058 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1246 is superior to 1024 2017-10-03 21:51:20,100 INFO [main] hbase.ResourceChecker(148): before: coprocessor.TestRegionObserverInterface#testAppendHook Thread=318, OpenFileDescriptor=1246, MaxFileDescriptor=32000, SystemLoadAverage=361, ProcessCount=347, AvailableMemoryMB=48672 2017-10-03 21:51:20,100 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1246 is superior to 1024 2017-10-03 21:51:20,107 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$3(1748): Client=hbase//172.18.128.12 create 'TestTable.testAppendHook', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2017-10-03 21:51:20,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=73, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestTable.testAppendHook 2017-10-03 21:51:20,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=73 2017-10-03 21:51:20,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=73 2017-10-03 21:51:20,401 INFO [IPC Server handler 7 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741884_1060{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-38a5a5c8-5f80-4892-8855-c1003ac8c597:NORMAL:127.0.0.1:41361|FINALIZED]]} size 0 2017-10-03 21:51:20,405 DEBUG [ProcExecWrkr-13] util.FSTableDescriptors(728): Wrote descriptor into: hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testAppendHook/.tabledesc/.tableinfo.0000000001 2017-10-03 21:51:20,408 INFO [RegionOpenAndInitThread-TestTable.testAppendHook-1] regionserver.HRegion(6477): creating HRegion TestTable.testAppendHook HTD == 'TestTable.testAppendHook', {NAME => 'a', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'b', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'c', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp Table name == TestTable.testAppendHook 2017-10-03 21:51:20,420 INFO [IPC Server handler 5 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741885_1061{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|FINALIZED]]} size 0 2017-10-03 21:51:20,421 DEBUG [RegionOpenAndInitThread-TestTable.testAppendHook-1] regionserver.HRegion(798): Instantiated TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. 2017-10-03 21:51:20,421 DEBUG [RegionOpenAndInitThread-TestTable.testAppendHook-1] regionserver.HRegion(1560): Closing TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1.: disabling compactions & flushes 2017-10-03 21:51:20,421 DEBUG [RegionOpenAndInitThread-TestTable.testAppendHook-1] regionserver.HRegion(1599): Updates disabled for region TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. 2017-10-03 21:51:20,421 INFO [RegionOpenAndInitThread-TestTable.testAppendHook-1] regionserver.HRegion(1712): Closed TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. 2017-10-03 21:51:20,530 DEBUG [ProcExecWrkr-13] hbase.MetaTableAccessor(1417): Put{"totalColumns":1,"row":"TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1.","families":{"info":[{"qualifier":"regioninfo","vlen":58,"tag":[],"timestamp":9223372036854775807}]}} 2017-10-03 21:51:20,535 INFO [ProcExecWrkr-13] hbase.MetaTableAccessor(1614): Added 1 2017-10-03 21:51:20,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=73 2017-10-03 21:51:20,611 DEBUG [ProcExecWrkr-13] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testAppendHook","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067480611}]}} 2017-10-03 21:51:20,616 INFO [ProcExecWrkr-13] hbase.MetaTableAccessor(1770): Updated table TestTable.testAppendHook state to ENABLING in META 2017-10-03 21:51:20,616 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=74, ppid=73, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testAppendHook, region=aa618fa2bbde29bb4fb2ceb04443a8b1}] 2017-10-03 21:51:20,687 INFO [ProcExecWrkr-13] procedure.MasterProcedureScheduler(814): pid=74, ppid=73, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testAppendHook, region=aa618fa2bbde29bb4fb2ceb04443a8b1 TestTable.testAppendHook TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. 2017-10-03 21:51:20,690 INFO [ProcExecWrkr-13] assignment.AssignProcedure(204): Start pid=74, ppid=73, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=TestTable.testAppendHook, region=aa618fa2bbde29bb4fb2ceb04443a8b1; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2017-10-03 21:51:20,842 INFO [ProcExecWrkr-20] assignment.RegionStateStore(186): pid=74 updating hbase:meta row=TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1., regionState=OPENING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:20,851 INFO [ProcExecWrkr-20] assignment.RegionTransitionProcedure(213): Dispatch pid=74, ppid=73, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testAppendHook, region=aa618fa2bbde29bb4fb2ceb04443a8b1; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:20,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=73 2017-10-03 21:51:20,915 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 44110ms 2017-10-03 21:51:20,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 260184ms 2017-10-03 21:51:21,001 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:51:21,003 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1916): Open TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. 2017-10-03 21:51:21,010 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(6654): Opening region: {ENCODED => aa618fa2bbde29bb4fb2ceb04443a8b1, NAME => 'TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1.', STARTKEY => '', ENDKEY => ''} 2017-10-03 21:51:21,011 INFO [RS_OPEN_REGION-cn012:37573-2] coprocessor.CoprocessorHost(164): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2017-10-03 21:51:21,011 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table TestTable.testAppendHook aa618fa2bbde29bb4fb2ceb04443a8b1 2017-10-03 21:51:21,011 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(798): Instantiated TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. 2017-10-03 21:51:21,017 DEBUG [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/a 2017-10-03 21:51:21,017 DEBUG [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/a 2017-10-03 21:51:21,018 INFO [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:51:21,018 INFO [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:51:21,018 INFO [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] hfile.CacheConfig(258): Created cacheConfig for a: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:51:21,019 INFO [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:51:21,024 DEBUG [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/b 2017-10-03 21:51:21,024 DEBUG [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/b 2017-10-03 21:51:21,024 INFO [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:51:21,024 INFO [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:51:21,025 INFO [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] hfile.CacheConfig(258): Created cacheConfig for b: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:51:21,025 INFO [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:51:21,030 DEBUG [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/c 2017-10-03 21:51:21,030 DEBUG [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/c 2017-10-03 21:51:21,031 INFO [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] regionserver.CompactingMemStore(138): Setting in-memory flush size threshold to 11184810 2017-10-03 21:51:21,031 INFO [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] regionserver.HStore(288): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-10-03 21:51:21,031 INFO [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] hfile.CacheConfig(258): Created cacheConfig for c: blockCache=LruBlockCache{blockCount=1, currentSize=766464, freeSize=1043195840, maxSize=1043962304, heapSize=766464, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-10-03 21:51:21,031 INFO [StoreOpener-aa618fa2bbde29bb4fb2ceb04443a8b1-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-10-03 21:51:21,033 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(1030): Setting FlushNonSloppyStoresFirstPolicy for the region=TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. 2017-10-03 21:51:21,034 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(4121): Found 0 recovered edits file(s) under hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1 2017-10-03 21:51:21,036 DEBUG [RS_OPEN_REGION-cn012:37573-2] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table TestTable.testAppendHook, use config (44739242) instead 2017-10-03 21:51:21,039 DEBUG [RS_OPEN_REGION-cn012:37573-2] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2017-10-03 21:51:21,039 INFO [RS_OPEN_REGION-cn012:37573-2] regionserver.HRegion(957): Onlined aa618fa2bbde29bb4fb2ceb04443a8b1; next sequenceid=2 2017-10-03 21:51:21,041 INFO [PostOpenDeployTasks:aa618fa2bbde29bb4fb2ceb04443a8b1] regionserver.HRegionServer(2170): Post open deploy tasks for TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. 2017-10-03 21:51:21,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report OPENED seqId=2, pid=74, ppid=73, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=TestTable.testAppendHook, region=aa618fa2bbde29bb4fb2ceb04443a8b1; rit=OPENING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:21,044 INFO [ProcExecWrkr-18] assignment.RegionStateStore(186): pid=74 updating hbase:meta row=TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1., regionState=OPEN, openSeqNum=2, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:21,044 DEBUG [PostOpenDeployTasks:aa618fa2bbde29bb4fb2ceb04443a8b1] regionserver.HRegionServer(2197): Finished post open deploy task for TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. 2017-10-03 21:51:21,044 DEBUG [RS_OPEN_REGION-cn012:37573-2] handler.OpenRegionHandler(127): Opened TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. on cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:21,115 INFO [ProcExecWrkr-18] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=74, ppid=73, state=SUCCESS; AssignProcedure table=TestTable.testAppendHook, region=aa618fa2bbde29bb4fb2ceb04443a8b1 2017-10-03 21:51:21,163 INFO [ProcExecWrkr-18] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=73, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=TestTable.testAppendHook; resume parent processing. 2017-10-03 21:51:21,164 INFO [ProcExecWrkr-18] procedure2.ProcedureExecutor(1246): Finished pid=74, ppid=73, state=SUCCESS; AssignProcedure table=TestTable.testAppendHook, region=aa618fa2bbde29bb4fb2ceb04443a8b1 in 432msec 2017-10-03 21:51:21,164 DEBUG [ProcExecWrkr-9] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testAppendHook","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067481164}]}} 2017-10-03 21:51:21,168 INFO [ProcExecWrkr-9] hbase.MetaTableAccessor(1770): Updated table TestTable.testAppendHook state to ENABLED in META 2017-10-03 21:51:21,287 INFO [ProcExecWrkr-9] procedure2.ProcedureExecutor(1246): Finished pid=73, state=SUCCESS; CreateTableProcedure table=TestTable.testAppendHook in 1.1240sec 2017-10-03 21:51:21,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=73 2017-10-03 21:51:21,406 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: CREATE, Table Name: default:TestTable.testAppendHook completed 2017-10-03 21:51:21,407 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:51:21,412 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-10-03 21:51:21,425 INFO [Time-limited test] client.HBaseAdmin$15(919): Started disable of TestTable.testAppendHook 2017-10-03 21:51:21,425 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$11(2331): Client=hbase//172.18.128.12 disable TestTable.testAppendHook 2017-10-03 21:51:21,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=75, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestTable.testAppendHook 2017-10-03 21:51:21,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=75 2017-10-03 21:51:21,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=75 2017-10-03 21:51:21,776 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2017-10-03 21:51:21,813 DEBUG [ProcExecWrkr-19] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testAppendHook","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067481812}]}} 2017-10-03 21:51:21,818 INFO [ProcExecWrkr-19] hbase.MetaTableAccessor(1770): Updated table TestTable.testAppendHook state to DISABLING in META 2017-10-03 21:51:21,879 INFO [ProcExecWrkr-19] procedure2.ProcedureExecutor(1495): Initialized subprocedures=[{pid=76, ppid=75, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testAppendHook, region=aa618fa2bbde29bb4fb2ceb04443a8b1, server=cn012.l42scl.hortonworks.com,37573,1507067090360}] 2017-10-03 21:51:21,950 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 41118ms 2017-10-03 21:51:21,951 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 144391ms 2017-10-03 21:51:21,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=75 2017-10-03 21:51:21,981 INFO [ProcExecWrkr-19] procedure.MasterProcedureScheduler(814): pid=76, ppid=75, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testAppendHook, region=aa618fa2bbde29bb4fb2ceb04443a8b1, server=cn012.l42scl.hortonworks.com,37573,1507067090360 TestTable.testAppendHook TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. 2017-10-03 21:51:21,981 INFO [ProcExecWrkr-19] assignment.RegionStateStore(186): pid=76 updating hbase:meta row=TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1., regionState=CLOSING, regionLocation=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:21,987 INFO [ProcExecWrkr-19] assignment.RegionTransitionProcedure(213): Dispatch pid=76, ppid=75, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testAppendHook, region=aa618fa2bbde29bb4fb2ceb04443a8b1, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:22,137 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=cn012.l42scl.hortonworks.com,37573,1507067090360 version=2097152 2017-10-03 21:51:22,139 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=37573] regionserver.RSRpcServices(1520): Close aa618fa2bbde29bb4fb2ceb04443a8b1 without moving 2017-10-03 21:51:22,140 DEBUG [RS_CLOSE_REGION-cn012:37573-0] handler.CloseRegionHandler(91): Processing close of TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. 2017-10-03 21:51:22,142 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(1560): Closing TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1.: disabling compactions & flushes 2017-10-03 21:51:22,142 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(1599): Updates disabled for region TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. 2017-10-03 21:51:22,142 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(2535): Flushing 3/3 column families, memstore=27 B 2017-10-03 21:51:22,142 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1.store: a 2017-10-03 21:51:22,143 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:51:22,143 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1.store: b 2017-10-03 21:51:22,143 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:0, and the number of cells in new segment is:0 2017-10-03 21:51:22,143 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1.store: c 2017-10-03 21:51:22,143 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:0, and the number of cells in new segment is:0 2017-10-03 21:51:22,164 INFO [IPC Server handler 2 on 35744] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41361 is added to blk_1073741886_1062{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-525133bb-b63e-46a5-9530-ae98802ccefb:NORMAL:127.0.0.1:41361|RBW]]} size 4660 2017-10-03 21:51:22,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=75 2017-10-03 21:51:22,388 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2017-10-03 21:51:22,406 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@70efb718] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:41361 to delete [blk_1073741881_1057, blk_1073741882_1058] 2017-10-03 21:51:22,566 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=5, memsize=27, hasBloomFilter=false, into tmp file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/.tmp/a/76f7e60ad18749129e80aecec79c6a94 2017-10-03 21:51:22,577 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/.tmp/a/76f7e60ad18749129e80aecec79c6a94 as hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/a/76f7e60ad18749129e80aecec79c6a94 2017-10-03 21:51:22,586 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HStore(1022): Added hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/a/76f7e60ad18749129e80aecec79c6a94, entries=1, sequenceid=5, filesize=4.6 K 2017-10-03 21:51:22,588 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(2731): Finished memstore flush of ~27 B/27, currentsize=0 B/0 for region TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. in 446ms, sequenceid=5, compaction requested=false 2017-10-03 21:51:22,590 INFO [StoreCloserThread-TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1.-1] regionserver.HStore(926): Closed a 2017-10-03 21:51:22,590 INFO [StoreCloserThread-TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1.-1] regionserver.HStore(926): Closed b 2017-10-03 21:51:22,591 INFO [StoreCloserThread-TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1.-1] regionserver.HStore(926): Closed c 2017-10-03 21:51:22,597 DEBUG [RS_CLOSE_REGION-cn012:37573-0] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/recovered.edits/8.seqid, newSeqId=8, maxSeqId=2 2017-10-03 21:51:22,597 DEBUG [RS_CLOSE_REGION-cn012:37573-0] coprocessor.CoprocessorHost(291): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2017-10-03 21:51:22,598 INFO [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(1712): Closed TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. 2017-10-03 21:51:22,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] assignment.RegionTransitionProcedure(235): Received report CLOSED seqId=-1, pid=76, ppid=75, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=TestTable.testAppendHook, region=aa618fa2bbde29bb4fb2ceb04443a8b1, server=cn012.l42scl.hortonworks.com,37573,1507067090360; rit=CLOSING, location=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:22,600 DEBUG [RS_CLOSE_REGION-cn012:37573-0] handler.CloseRegionHandler(123): Closed TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. 2017-10-03 21:51:22,600 INFO [ProcExecWrkr-23] assignment.RegionStateStore(186): pid=76 updating hbase:meta row=TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1., regionState=CLOSED 2017-10-03 21:51:22,645 INFO [ProcExecWrkr-23] procedure2.ProcedureExecutor(1609): Finish suprocedure pid=76, ppid=75, state=SUCCESS; UnassignProcedure table=TestTable.testAppendHook, region=aa618fa2bbde29bb4fb2ceb04443a8b1, server=cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:22,695 INFO [ProcExecWrkr-23] procedure2.ProcedureExecutor(1615): Finished subprocedure(s) of pid=75, state=RUNNABLE:DISABLE_TABLE_SET_DISABLED_TABLE_STATE; DisableTableProcedure table=TestTable.testAppendHook; resume parent processing. 2017-10-03 21:51:22,696 DEBUG [ProcExecWrkr-22] hbase.MetaTableAccessor(1398): Put{"totalColumns":1,"row":"TestTable.testAppendHook","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1507067482695}]}} 2017-10-03 21:51:22,696 INFO [ProcExecWrkr-23] procedure2.ProcedureExecutor(1246): Finished pid=76, ppid=75, state=SUCCESS; UnassignProcedure table=TestTable.testAppendHook, region=aa618fa2bbde29bb4fb2ceb04443a8b1, server=cn012.l42scl.hortonworks.com,37573,1507067090360 in 724msec 2017-10-03 21:51:22,699 INFO [ProcExecWrkr-22] hbase.MetaTableAccessor(1770): Updated table TestTable.testAppendHook state to DISABLED in META 2017-10-03 21:51:22,699 INFO [ProcExecWrkr-22] procedure.DisableTableProcedure(285): Disabled table, TestTable.testAppendHook, is completed. 2017-10-03 21:51:22,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=75 2017-10-03 21:51:22,776 ERROR [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] ipc.RpcServer(464): Unexpected throwable object java.lang.AssertionError: pid=75 is still running at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.removeResult(ProcedureExecutor.java:1012) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:1155) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-10-03 21:51:22,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] ipc.CallRunner(144): callId: 578 service: MasterService methodName: getProcedureResult size: 35 connection: 172.18.128.12:40750 deadline: 1507067542776 java.io.IOException: pid=75 is still running at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:465) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) Caused by: java.lang.AssertionError: pid=75 is still running at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.removeResult(ProcedureExecutor.java:1012) at org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:1155) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) ... 3 more 2017-10-03 21:51:22,795 INFO [ProcExecWrkr-22] procedure2.ProcedureExecutor(1246): Finished pid=75, state=SUCCESS; DisableTableProcedure table=TestTable.testAppendHook in 1.3190sec 2017-10-03 21:51:22,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=75 2017-10-03 21:51:22,907 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DISABLE, Table Name: default:TestTable.testAppendHook completed 2017-10-03 21:51:22,909 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.HMaster$5(2109): Client=hbase//172.18.128.12 delete TestTable.testAppendHook 2017-10-03 21:51:22,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. because info has an old edit so flush to free WALs after random delay 270074ms 2017-10-03 21:51:22,916 INFO [cn012.l42scl.hortonworks.com,37573,1507067090360_Chore_1] regionserver.HRegionServer$PeriodicMemStoreFlusher(1830): MemstoreFlusherChore requesting flush of hbase:meta,,1.1588230740 because info has an old edit so flush to free WALs after random delay 235582ms 2017-10-03 21:51:23,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] procedure2.ProcedureExecutor(868): Stored pid=77, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestTable.testAppendHook 2017-10-03 21:51:23,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=77 2017-10-03 21:51:23,069 DEBUG [ProcExecWrkr-24] procedure.DeleteTableProcedure(99): Waiting for 'TestTable.testAppendHook' regions in transition 2017-10-03 21:51:23,129 DEBUG [ProcExecWrkr-24] procedure.DeleteTableProcedure(110): delete 'TestTable.testAppendHook' regions from META 2017-10-03 21:51:23,130 DEBUG [ProcExecWrkr-24] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067483129}]},"ts":9223372036854775807} 2017-10-03 21:51:23,136 INFO [ProcExecWrkr-24] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:51:23,136 DEBUG [ProcExecWrkr-24] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => aa618fa2bbde29bb4fb2ceb04443a8b1, NAME => 'TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:51:23,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=77 2017-10-03 21:51:23,212 DEBUG [ProcExecWrkr-24] procedure.DeleteTableProcedure(115): delete 'TestTable.testAppendHook' from filesystem 2017-10-03 21:51:23,217 DEBUG [ProcExecWrkr-24] procedure.DeleteTableProcedure(315): Archiving region TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1. from FS 2017-10-03 21:51:23,217 DEBUG [ProcExecWrkr-24] backup.HFileArchiver(114): ARCHIVING hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1 2017-10-03 21:51:23,223 DEBUG [ProcExecWrkr-24] backup.HFileArchiver(155): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/recovered.edits] 2017-10-03 21:51:23,231 DEBUG [ProcExecWrkr-24] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/a/76f7e60ad18749129e80aecec79c6a94, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/a/76f7e60ad18749129e80aecec79c6a94 2017-10-03 21:51:23,239 DEBUG [ProcExecWrkr-24] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/recovered.edits/8.seqid, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1/recovered.edits/8.seqid 2017-10-03 21:51:23,239 INFO [IPC Server handler 5 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741885_1061 127.0.0.1:41361 2017-10-03 21:51:23,240 DEBUG [ProcExecWrkr-24] backup.HFileArchiver(466): Deleted hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/.tmp/data/default/TestTable.testAppendHook/aa618fa2bbde29bb4fb2ceb04443a8b1 2017-10-03 21:51:23,240 DEBUG [ProcExecWrkr-24] procedure.DeleteTableProcedure(319): Table 'TestTable.testAppendHook' archived! 2017-10-03 21:51:23,241 INFO [IPC Server handler 1 on 35744] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741884_1060 127.0.0.1:41361 2017-10-03 21:51:23,303 DEBUG [ProcExecWrkr-24] procedure.DeleteTableProcedure(121): delete 'TestTable.testAppendHook' descriptor 2017-10-03 21:51:23,304 DEBUG [ProcExecWrkr-24] procedure.DeleteTableProcedure(399): Removing 'TestTable.testAppendHook' descriptor. 2017-10-03 21:51:23,362 DEBUG [ProcExecWrkr-24] procedure.DeleteTableProcedure(126): delete 'TestTable.testAppendHook' assignment state 2017-10-03 21:51:23,362 DEBUG [ProcExecWrkr-24] procedure.DeleteTableProcedure(389): Removing 'TestTable.testAppendHook' from region states. 2017-10-03 21:51:23,363 DEBUG [ProcExecWrkr-24] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1507067483362}]},"ts":9223372036854775807} 2017-10-03 21:51:23,366 INFO [ProcExecWrkr-24] hbase.MetaTableAccessor(1983): Deleted 1 regions from META 2017-10-03 21:51:23,367 DEBUG [ProcExecWrkr-24] hbase.MetaTableAccessor(1985): Deleted regions: [{ENCODED => aa618fa2bbde29bb4fb2ceb04443a8b1, NAME => 'TestTable.testAppendHook,,1507067480107.aa618fa2bbde29bb4fb2ceb04443a8b1.', STARTKEY => '', ENDKEY => ''}] 2017-10-03 21:51:23,367 DEBUG [ProcExecWrkr-24] procedure.DeleteTableProcedure(393): Marking 'TestTable.testAppendHook' as deleted. 2017-10-03 21:51:23,367 DEBUG [ProcExecWrkr-24] hbase.MetaTableAccessor(1449): Delete{"totalColumns":1,"row":"TestTable.testAppendHook","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":1507067483367}]},"ts":9223372036854775807} 2017-10-03 21:51:23,369 INFO [ProcExecWrkr-24] hbase.MetaTableAccessor(1796): Deleted table TestTable.testAppendHook state from META 2017-10-03 21:51:23,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=77 2017-10-03 21:51:23,421 DEBUG [ProcExecWrkr-24] procedure.DeleteTableProcedure(132): delete 'TestTable.testAppendHook' completed 2017-10-03 21:51:23,487 INFO [ProcExecWrkr-24] procedure2.ProcedureExecutor(1246): Finished pid=77, state=SUCCESS; DeleteTableProcedure table=TestTable.testAppendHook in 512msec 2017-10-03 21:51:23,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=35273] master.MasterRpcServices(1124): Checking to see if procedure is done pid=77 2017-10-03 21:51:23,674 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3709): Operation: DELETE, Table Name: default:TestTable.testAppendHook completed 2017-10-03 21:51:23,709 INFO [main] hbase.ResourceChecker(172): after: coprocessor.TestRegionObserverInterface#testAppendHook Thread=327 (was 318) - Thread LEAK? -, OpenFileDescriptor=1246 (was 1246), MaxFileDescriptor=32000 (was 32000), SystemLoadAverage=356 (was 361), ProcessCount=349 (was 347) - ProcessCount LEAK? -, AvailableMemoryMB=49468 (was 48672) - AvailableMemoryMB LEAK? - 2017-10-03 21:51:23,709 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1246 is superior to 1024 2017-10-03 21:51:23,709 INFO [main] hbase.HBaseTestingUtility(1189): Shutting down minicluster 2017-10-03 21:51:23,709 INFO [main] client.ConnectionImplementation(1828): Closing master protocol: MasterService 2017-10-03 21:51:23,710 INFO [main] client.ConnectionImplementation(1247): Closing zookeeper sessionid=0x15ee432cd9b0007 2017-10-03 21:51:23,733 DEBUG [main] ipc.AbstractRpcClient(483): Stopping rpc client 2017-10-03 21:51:23,734 DEBUG [main] util.JVMClusterUtil(246): Shutting down HBase Cluster 2017-10-03 21:51:23,734 INFO [main] master.ServerManager(947): Cluster shutdown requested of master=cn012.l42scl.hortonworks.com,35273,1507067089134 2017-10-03 21:51:23,734 INFO [main] regionserver.HRegionServer(2126): ***** STOPPING region server 'cn012.l42scl.hortonworks.com,35273,1507067089134' ***** 2017-10-03 21:51:23,734 INFO [main] regionserver.HRegionServer(2140): STOPPED: Cluster shutdown requested of master=cn012.l42scl.hortonworks.com,35273,1507067089134 2017-10-03 21:51:23,735 INFO [M:0;cn012:35273] regionserver.HRegionServer(1150): stopping server cn012.l42scl.hortonworks.com,35273,1507067089134 2017-10-03 21:51:23,735 DEBUG [M:0;cn012:35273] zookeeper.MetaTableLocator(625): Stopping MetaTableLocator 2017-10-03 21:51:23,736 INFO [M:0;cn012:35273] client.ConnectionImplementation(1247): Closing zookeeper sessionid=0x15ee432cd9b0002 2017-10-03 21:51:23,773 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2017-10-03 21:51:23,773 INFO [main] regionserver.HRegionServer(2126): ***** STOPPING region server 'cn012.l42scl.hortonworks.com,37573,1507067090360' ***** 2017-10-03 21:51:23,773 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2017-10-03 21:51:23,774 INFO [main] regionserver.HRegionServer(2140): STOPPED: Shutdown requested 2017-10-03 21:51:23,774 INFO [RS:0;cn012:37573] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2017-10-03 21:51:23,774 INFO [RS:0;cn012:37573] regionserver.HeapMemoryManager(223): Stopping HeapMemoryTuner chore. 2017-10-03 21:51:23,775 INFO [SplitLogWorker-cn012:37573] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2017-10-03 21:51:23,775 INFO [SplitLogWorker-cn012:37573] regionserver.SplitLogWorker(155): SplitLogWorker cn012.l42scl.hortonworks.com,37573,1507067090360 exiting 2017-10-03 21:51:23,775 INFO [RS:0;cn012:37573] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2017-10-03 21:51:23,776 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(275): MemStoreFlusher.1 exiting 2017-10-03 21:51:23,776 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(275): MemStoreFlusher.0 exiting 2017-10-03 21:51:23,776 INFO [RS:0;cn012:37573] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2017-10-03 21:51:23,777 DEBUG [RS_CLOSE_REGION-cn012:37573-1] handler.CloseRegionHandler(91): Processing close of hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. 2017-10-03 21:51:23,777 DEBUG [RS_CLOSE_REGION-cn012:37573-0] handler.CloseRegionHandler(91): Processing close of testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. 2017-10-03 21:51:23,777 DEBUG [RS_CLOSE_REGION-cn012:37573-2] handler.CloseRegionHandler(91): Processing close of TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:51:23,777 INFO [RS:0;cn012:37573] regionserver.HRegionServer(1150): stopping server cn012.l42scl.hortonworks.com,37573,1507067090360 2017-10-03 21:51:23,779 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(1560): Closing TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.: disabling compactions & flushes 2017-10-03 21:51:23,779 DEBUG [RS:0;cn012:37573] zookeeper.MetaTableLocator(625): Stopping MetaTableLocator 2017-10-03 21:51:23,779 INFO [RS:0;cn012:37573] client.ConnectionImplementation(1247): Closing zookeeper sessionid=0x15ee432cd9b0003 2017-10-03 21:51:23,778 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(1560): Closing testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781.: disabling compactions & flushes 2017-10-03 21:51:23,778 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(1560): Closing hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3.: disabling compactions & flushes 2017-10-03 21:51:23,779 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(1599): Updates disabled for region hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3. 2017-10-03 21:51:23,779 DEBUG [RS_CLOSE_REGION-cn012:37573-0] regionserver.HRegion(1599): Updates disabled for region testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781. 2017-10-03 21:51:23,779 DEBUG [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(1599): Updates disabled for region TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:51:23,779 INFO [RS_CLOSE_REGION-cn012:37573-1] regionserver.HRegion(2535): Flushing 1/1 column families, memstore=78 B 2017-10-03 21:51:23,780 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.CompactingMemStore(195): FLUSHING TO DISK: region hbase:namespace,,1507067094474.ae72b836f7d2128d19490a1bda8b68a3.store: info 2017-10-03 21:51:23,784 DEBUG [StoreCloserThread-testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781.-1] regionserver.HStore(2549): Moving the files [hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781/a/166bdda8b1124a1d86c3ca69d46fc310] to archive 2017-10-03 21:51:23,784 INFO [StoreCloserThread-TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.-1] regionserver.HStore(926): Closed a 2017-10-03 21:51:23,784 DEBUG [RS_CLOSE_REGION-cn012:37573-1] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-10-03 21:51:23,784 INFO [StoreCloserThread-TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.-1] regionserver.HStore(926): Closed b 2017-10-03 21:51:23,784 INFO [StoreCloserThread-TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.-1] regionserver.HStore(926): Closed c 2017-10-03 21:51:23,787 DEBUG [StoreCloserThread-testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781.-1] backup.HFileArchiver(259): Archiving compacted store files. 2017-10-03 21:51:23,789 WARN [StoreCloserThread-testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781.-1] hdfs.DFSInputStream(669): DFSInputStream has been closed already 2017-10-03 21:51:23,791 DEBUG [RS_CLOSE_REGION-cn012:37573-2] wal.WALSplitter(736): Wrote file=hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/TestTable.testRecovery/e8d466bc47137a0d2d7b3119106dc88a/recovered.edits/11.seqid, newSeqId=11, maxSeqId=9 2017-10-03 21:51:23,792 DEBUG [RS_CLOSE_REGION-cn012:37573-2] coprocessor.CoprocessorHost(291): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2017-10-03 21:51:23,792 DEBUG [StoreCloserThread-testCompactionOverride,,1507067447885.338d120f0bd69660aa21ecb994fb9781.-1] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781/a/166bdda8b1124a1d86c3ca69d46fc310, to hdfs://localhost:35744/user/hbase/test-data/17afbae4-09d6-48a2-96cb-38c67e00fff0/archive/data/default/testCompactionOverride/338d120f0bd69660aa21ecb994fb9781/a/166bdda8b1124a1d86c3ca69d46fc310 2017-10-03 21:51:23,792 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegion(1712): Closed TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a. 2017-10-03 21:51:23,797 DEBUG [main-EventThread] zookeeper.ZKUtil(364): master:35273-0x15ee432cd9b0000, quorum=localhost:57257, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2017-10-03 21:51:23,797 DEBUG [main-EventThread] zookeeper.ZKUtil(364): regionserver:37573-0x15ee432cd9b0001, quorum=localhost:57257, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2017-10-03 21:51:23,797 DEBUG [M:0;cn012:35273] ipc.AbstractRpcClient(483): Stopping rpc client 2017-10-03 21:51:23,798 INFO [RS_CLOSE_REGION-cn012:37573-2] regionserver.HRegionServer(2303): TRANSITION NOT REPORTED server { host_name: "cn012.l42scl.hortonworks.com" port: 37573 start_code: 1507067090360 } transition { transition_code: CLOSED region_info { region_id: 1507067136028 table_name { namespace: "default" qualifier: "TestTable.testRecovery" } start_key: "" end_key: "" offline: false split: false replica_id: 0 } } 2017-10-03 21:51:23,798 DEBUG [RS_CLOSE_REGION-cn012:37573-2] handler.CloseRegionHandler(123): Closed TestTable.testRecovery,,1507067136028.e8d466bc47137a0d2d7b3119106dc88a.