2017-06-28 16:31:26,564 INFO [main] hbase.ResourceChecker(148): before: client.TestReplicaWithCluster#testReplicaGetWithPrimaryAndMetaDown Thread=567, OpenFileDescriptor=1379, MaxFileDescriptor=60000, SystemLoadAverage=287, ProcessCount=269, AvailableMemoryMB=17677 2017-06-28 16:31:26,565 WARN [main] hbase.ResourceChecker(135): Thread=567 is superior to 500 2017-06-28 16:31:26,565 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1379 is superior to 1024 2017-06-28 16:31:26,568 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:40948-0x15cef8a87ad0000, quorum=localhost:52259, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2017-06-28 16:31:26,568 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] master.MasterRpcServices(234): Client=jenkins//67.195.81.146 set balanceSwitch=false 2017-06-28 16:31:26,571 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] master.HMaster$4(1764): Client=jenkins//67.195.81.146 create 'testReplicaGetWithPrimaryAndMetaDown', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '2'}, {NAME => 'colfamily11', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'false', BLOCKSIZE => '65536'}, {NAME => 'colfamily21', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'false', BLOCKSIZE => '65536'}, {NAME => 'colfamily31', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'false', BLOCKSIZE => '65536'}, {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2017-06-28 16:31:26,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] procedure2.ProcedureExecutor(792): Stored pid=55, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testReplicaGetWithPrimaryAndMetaDown 2017-06-28 16:31:26,677 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(186): Get, We're replica region 0 2017-06-28 16:31:26,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] master.MasterRpcServices(1011): Checking to see if procedure is done procId=55 2017-06-28 16:31:26,695 INFO [IPC Server handler 8 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40636 is added to blk_1073741858_1034{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-aaac77f0-5ffc-458f-8598-c9877d325015:NORMAL:127.0.0.1:43805|RBW], ReplicaUC[[DISK]DS-94f4b1e2-c0c4-46fb-8b5d-f3d3d203b341:NORMAL:127.0.0.1:36908|RBW], ReplicaUC[[DISK]DS-518f10dc-7009-49cd-8bb9-88e593430f4f:NORMAL:127.0.0.1:40636|RBW]]} size 0 2017-06-28 16:31:26,696 INFO [IPC Server handler 3 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36908 is added to blk_1073741858_1034{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-aaac77f0-5ffc-458f-8598-c9877d325015:NORMAL:127.0.0.1:43805|RBW], ReplicaUC[[DISK]DS-518f10dc-7009-49cd-8bb9-88e593430f4f:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-db41de19-abcb-4f8b-9965-8094f7de5347:NORMAL:127.0.0.1:36908|FINALIZED]]} size 0 2017-06-28 16:31:26,696 INFO [IPC Server handler 2 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43805 is added to blk_1073741858_1034{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-aaac77f0-5ffc-458f-8598-c9877d325015:NORMAL:127.0.0.1:43805|RBW], ReplicaUC[[DISK]DS-518f10dc-7009-49cd-8bb9-88e593430f4f:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-db41de19-abcb-4f8b-9965-8094f7de5347:NORMAL:127.0.0.1:36908|FINALIZED]]} size 0 2017-06-28 16:31:26,699 DEBUG [ProcExecWrkr-7] util.FSTableDescriptors(702): Wrote descriptor into: hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/.tmp/data/default/testReplicaGetWithPrimaryAndMetaDown/.tabledesc/.tableinfo.0000000001 2017-06-28 16:31:26,701 INFO [RegionOpenAndInitThread-testReplicaGetWithPrimaryAndMetaDown-1] regionserver.HRegion(6536): creating HRegion testReplicaGetWithPrimaryAndMetaDown HTD == 'testReplicaGetWithPrimaryAndMetaDown', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '2'}, {NAME => 'colfamily11', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'false', BLOCKSIZE => '65536'}, {NAME => 'colfamily21', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'false', BLOCKSIZE => '65536'}, {NAME => 'colfamily31', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'false', BLOCKSIZE => '65536'}, {NAME => 'info', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'false', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/.tmp Table name == testReplicaGetWithPrimaryAndMetaDown 2017-06-28 16:31:26,714 INFO [IPC Server handler 3 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43805 is added to blk_1073741859_1035{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-758fa1ec-cf5f-4f33-8dd8-a9ce20d4953e:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-94f4b1e2-c0c4-46fb-8b5d-f3d3d203b341:NORMAL:127.0.0.1:36908|RBW], ReplicaUC[[DISK]DS-75b4545b-7081-4bb7-8c67-5807d44e67e0:NORMAL:127.0.0.1:43805|RBW]]} size 0 2017-06-28 16:31:26,714 INFO [IPC Server handler 2 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36908 is added to blk_1073741859_1035{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-758fa1ec-cf5f-4f33-8dd8-a9ce20d4953e:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-94f4b1e2-c0c4-46fb-8b5d-f3d3d203b341:NORMAL:127.0.0.1:36908|RBW], ReplicaUC[[DISK]DS-75b4545b-7081-4bb7-8c67-5807d44e67e0:NORMAL:127.0.0.1:43805|RBW]]} size 0 2017-06-28 16:31:26,714 INFO [IPC Server handler 8 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40636 is added to blk_1073741859_1035{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-758fa1ec-cf5f-4f33-8dd8-a9ce20d4953e:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-94f4b1e2-c0c4-46fb-8b5d-f3d3d203b341:NORMAL:127.0.0.1:36908|RBW], ReplicaUC[[DISK]DS-75b4545b-7081-4bb7-8c67-5807d44e67e0:NORMAL:127.0.0.1:43805|RBW]]} size 0 2017-06-28 16:31:26,715 DEBUG [RegionOpenAndInitThread-testReplicaGetWithPrimaryAndMetaDown-1] regionserver.HRegion(796): Instantiated testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:26,716 DEBUG [RegionOpenAndInitThread-testReplicaGetWithPrimaryAndMetaDown-1] regionserver.HRegion(1573): Closing testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.: disabling compactions & flushes 2017-06-28 16:31:26,716 DEBUG [RegionOpenAndInitThread-testReplicaGetWithPrimaryAndMetaDown-1] regionserver.HRegion(1612): Updates disabled for region testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:26,716 INFO [RegionOpenAndInitThread-testReplicaGetWithPrimaryAndMetaDown-1] regionserver.HRegion(1725): Closed testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:26,724 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(1410): Put{"totalColumns":4,"row":"testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":9223372036854775807},{"qualifier":"server_0001","vlen":0,"tag":[],"timestamp":1498667486723},{"qualifier":"serverstartcode_0001","vlen":0,"tag":[],"timestamp":1498667486723},{"qualifier":"seqnumDuringOpen_0001","vlen":0,"tag":[],"timestamp":1498667486723}]}} 2017-06-28 16:31:26,729 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1607): Added 1 2017-06-28 16:31:26,731 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(1391): Put{"totalColumns":1,"row":"testReplicaGetWithPrimaryAndMetaDown","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1498667486731}]}} 2017-06-28 16:31:26,735 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1767): Updated table testReplicaGetWithPrimaryAndMetaDown state to ENABLING in META 2017-06-28 16:31:26,735 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1416): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b}, {pid=57, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697}] 2017-06-28 16:31:26,744 INFO [ProcExecWrkr-5] procedure.MasterProcedureScheduler(822): pid=57, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697 testReplicaGetWithPrimaryAndMetaDown testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:26,744 INFO [ProcExecWrkr-6] procedure.MasterProcedureScheduler(822): pid=56, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b testReplicaGetWithPrimaryAndMetaDown testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:26,744 INFO [ProcExecWrkr-5] assignment.AssignProcedure(192): Start pid=57, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2017-06-28 16:31:26,744 INFO [ProcExecWrkr-6] assignment.AssignProcedure(192): Start pid=56, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2017-06-28 16:31:26,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40948] master.MasterRpcServices(1011): Checking to see if procedure is done procId=55 2017-06-28 16:31:26,895 DEBUG [AssignmentThread] balancer.RegionLocationFinder(282): HDFSBlocksDistribution not found in cache for region testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:26,903 DEBUG [AssignmentThread] balancer.RegionLocationFinder(282): HDFSBlocksDistribution not found in cache for region testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:26,910 INFO [ProcExecWrkr-3] assignment.RegionStateStore(186): pid=57 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697., regionState=OPENING, regionLocation=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:26,910 INFO [ProcExecWrkr-4] assignment.RegionStateStore(186): pid=56 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., regionState=OPENING, regionLocation=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:26,915 INFO [ProcExecWrkr-3] assignment.RegionTransitionProcedure(179): Dispatch pid=57, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697; rit=OPENING, location=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:26,916 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(179): Dispatch pid=56, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b; rit=OPENING, location=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:26,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40948] master.MasterRpcServices(1011): Checking to see if procedure is done procId=55 2017-06-28 16:31:27,066 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=asf902.gq1.ygridcore.net,53897,1498667454037 version=3145728 2017-06-28 16:31:27,066 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=asf902.gq1.ygridcore.net,58993,1498667454123 version=3145728 2017-06-28 16:31:27,067 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=53897] regionserver.RSRpcServices(1899): Open testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,067 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=58993] regionserver.RSRpcServices(1899): Open testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:27,073 DEBUG [RS_OPEN_REGION-asf902:58993-2] regionserver.HRegion(6713): Opening region: {ENCODED => da78e90b99801a7d916ba59be21ad54b, NAME => 'testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1} 2017-06-28 16:31:27,074 DEBUG [RS_OPEN_REGION-asf902:53897-1] regionserver.HRegion(6713): Opening region: {ENCODED => 1cf058a284bbd8c68aa7536052bfa697, NAME => 'testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.', STARTKEY => '', ENDKEY => ''} 2017-06-28 16:31:27,074 INFO [RS_OPEN_REGION-asf902:58993-2] coprocessor.CoprocessorHost(160): System coprocessor org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro was loaded successfully with priority (536870911). 2017-06-28 16:31:27,074 INFO [RS_OPEN_REGION-asf902:53897-1] coprocessor.CoprocessorHost(160): System coprocessor org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro was loaded successfully with priority (536870911). 2017-06-28 16:31:27,075 DEBUG [RS_OPEN_REGION-asf902:58993-2] regionserver.MetricsRegionSourceImpl(74): Creating new MetricsRegionSourceImpl for table testReplicaGetWithPrimaryAndMetaDown da78e90b99801a7d916ba59be21ad54b 2017-06-28 16:31:27,075 DEBUG [RS_OPEN_REGION-asf902:53897-1] regionserver.MetricsRegionSourceImpl(74): Creating new MetricsRegionSourceImpl for table testReplicaGetWithPrimaryAndMetaDown 1cf058a284bbd8c68aa7536052bfa697 2017-06-28 16:31:27,075 DEBUG [RS_OPEN_REGION-asf902:58993-2] regionserver.HRegion(796): Instantiated testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:27,075 DEBUG [RS_OPEN_REGION-asf902:53897-1] regionserver.HRegion(796): Instantiated testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,080 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily11 2017-06-28 16:31:27,080 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily11 2017-06-28 16:31:27,080 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily11 2017-06-28 16:31:27,080 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily11 2017-06-28 16:31:27,080 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,081 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,081 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,081 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,081 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] hfile.CacheConfig(258): Created cacheConfig for colfamily11: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,081 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] hfile.CacheConfig(258): Created cacheConfig for colfamily11: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,081 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,082 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,086 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily21 2017-06-28 16:31:27,086 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily21 2017-06-28 16:31:27,086 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily21 2017-06-28 16:31:27,087 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily21 2017-06-28 16:31:27,087 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,087 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,087 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,087 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,087 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] hfile.CacheConfig(258): Created cacheConfig for colfamily21: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,087 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] hfile.CacheConfig(258): Created cacheConfig for colfamily21: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,088 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,088 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,091 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily31 2017-06-28 16:31:27,092 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily31 2017-06-28 16:31:27,092 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily31 2017-06-28 16:31:27,092 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily31 2017-06-28 16:31:27,092 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,092 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,092 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,093 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,093 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] hfile.CacheConfig(258): Created cacheConfig for colfamily31: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,093 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] hfile.CacheConfig(258): Created cacheConfig for colfamily31: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,093 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,093 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,096 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info 2017-06-28 16:31:27,096 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info 2017-06-28 16:31:27,097 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info 2017-06-28 16:31:27,097 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info 2017-06-28 16:31:27,097 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,097 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,097 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,097 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,097 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] hfile.CacheConfig(258): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,098 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] hfile.CacheConfig(258): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,098 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,098 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,100 INFO [RS_OPEN_REGION-asf902:53897-1] regionserver.HRegion(1032): Setting FlushNonSloppyStoresFirstPolicy for the region=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,100 INFO [RS_OPEN_REGION-asf902:58993-2] regionserver.HRegion(1032): Setting FlushNonSloppyStoresFirstPolicy for the region=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:27,100 DEBUG [RS_OPEN_REGION-asf902:58993-2] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table testReplicaGetWithPrimaryAndMetaDown, use config (33554432) instead 2017-06-28 16:31:27,100 INFO [RS_OPEN_REGION-asf902:58993-2] regionserver.HRegion(959): Onlined da78e90b99801a7d916ba59be21ad54b; next sequenceid=2 2017-06-28 16:31:27,101 DEBUG [RS_OPEN_REGION-asf902:53897-1] regionserver.HRegion(4147): Found 0 recovered edits file(s) under hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697 2017-06-28 16:31:27,101 INFO [PostOpenDeployTasks:da78e90b99801a7d916ba59be21ad54b] regionserver.HRegionServer(2137): Post open deploy tasks for testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:27,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40948] assignment.RegionTransitionProcedure(200): Received report OPENED seqId=2, pid=56, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b; rit=OPENING, location=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:27,105 DEBUG [PostOpenDeployTasks:da78e90b99801a7d916ba59be21ad54b] regionserver.HRegionServer(2164): Finished post open deploy task for testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:27,105 INFO [ProcExecWrkr-13] assignment.RegionStateStore(186): pid=56 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., regionState=OPEN, openSeqNum=2, regionLocation=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:27,106 DEBUG [RS_OPEN_REGION-asf902:53897-1] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table testReplicaGetWithPrimaryAndMetaDown, use config (33554432) instead 2017-06-28 16:31:27,106 DEBUG [RS_OPEN_REGION-asf902:58993-2] handler.OpenRegionHandler(126): Opened testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. on asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:27,111 DEBUG [RS_OPEN_REGION-asf902:53897-1] wal.WALSplitter(736): Wrote file=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2017-06-28 16:31:27,111 INFO [RS_OPEN_REGION-asf902:53897-1] regionserver.HRegion(959): Onlined 1cf058a284bbd8c68aa7536052bfa697; next sequenceid=2 2017-06-28 16:31:27,114 INFO [PostOpenDeployTasks:1cf058a284bbd8c68aa7536052bfa697] regionserver.HRegionServer(2137): Post open deploy tasks for testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,114 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1530): Finish suprocedure pid=56, ppid=55, state=SUCCESS; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b 2017-06-28 16:31:27,114 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1167): Finished pid=56, ppid=55, state=SUCCESS; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b in 377msec 2017-06-28 16:31:27,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40948] assignment.RegionTransitionProcedure(200): Received report OPENED seqId=2, pid=57, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697; rit=OPENING, location=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:27,117 INFO [ProcExecWrkr-12] assignment.RegionStateStore(186): pid=57 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697., regionState=OPEN, openSeqNum=2, regionLocation=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:27,117 DEBUG [PostOpenDeployTasks:1cf058a284bbd8c68aa7536052bfa697] regionserver.HRegionServer(2164): Finished post open deploy task for testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,120 DEBUG [RS_OPEN_REGION-asf902:53897-1] handler.OpenRegionHandler(126): Opened testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. on asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:27,125 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1530): Finish suprocedure pid=57, ppid=55, state=SUCCESS; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697 2017-06-28 16:31:27,128 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1536): Finished subprocedure(s) of pid=55, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=testReplicaGetWithPrimaryAndMetaDown; resume parent processing. 2017-06-28 16:31:27,128 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1167): Finished pid=57, ppid=55, state=SUCCESS; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697 in 388msec 2017-06-28 16:31:27,129 DEBUG [ProcExecWrkr-8] hbase.MetaTableAccessor(1391): Put{"totalColumns":1,"row":"testReplicaGetWithPrimaryAndMetaDown","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1498667487129}]}} 2017-06-28 16:31:27,133 INFO [ProcExecWrkr-8] hbase.MetaTableAccessor(1767): Updated table testReplicaGetWithPrimaryAndMetaDown state to ENABLED in META 2017-06-28 16:31:27,138 INFO [ProcExecWrkr-8] procedure2.ProcedureExecutor(1167): Finished pid=55, state=SUCCESS; CreateTableProcedure table=testReplicaGetWithPrimaryAndMetaDown in 565msec 2017-06-28 16:31:27,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40948] master.MasterRpcServices(1011): Checking to see if procedure is done procId=55 2017-06-28 16:31:27,286 INFO [main] client.HBaseAdmin$TableFuture(3584): Operation: CREATE, Table Name: default:testReplicaGetWithPrimaryAndMetaDown completed 2017-06-28 16:31:27,286 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-06-28 16:31:27,288 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(215): Scan, We're replica region 0 2017-06-28 16:31:27,292 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2017-06-28 16:31:27,297 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(215): Scan, We're replica region 0 2017-06-28 16:31:27,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=58993] regionserver.RSRpcServices(2021): Warming up Region testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=58993] regionserver.HRegion(6780): HRegion.Warming up region: {ENCODED => 1cf058a284bbd8c68aa7536052bfa697, NAME => 'testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.', STARTKEY => '', ENDKEY => ''} 2017-06-28 16:31:27,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=58993] regionserver.HRegion(796): Instantiated testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,319 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily11 2017-06-28 16:31:27,319 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily11 2017-06-28 16:31:27,319 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,319 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,320 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] hfile.CacheConfig(258): Created cacheConfig for colfamily11: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,320 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,323 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily21 2017-06-28 16:31:27,323 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily21 2017-06-28 16:31:27,323 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,324 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,324 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] hfile.CacheConfig(258): Created cacheConfig for colfamily21: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,324 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,326 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily31 2017-06-28 16:31:27,326 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily31 2017-06-28 16:31:27,327 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,327 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,327 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] hfile.CacheConfig(258): Created cacheConfig for colfamily31: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,327 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,329 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info 2017-06-28 16:31:27,329 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info 2017-06-28 16:31:27,330 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,330 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,330 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] hfile.CacheConfig(258): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,330 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,331 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=58993] regionserver.HRegion(1032): Setting FlushNonSloppyStoresFirstPolicy for the region=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40948] master.HMaster(1724): Client=jenkins//67.195.81.146 move hri=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697., source=asf902.gq1.ygridcore.net,53897,1498667454037, destination=asf902.gq1.ygridcore.net,58993,1498667454123, running balancer 2017-06-28 16:31:27,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40948] procedure2.ProcedureExecutor(792): Stored pid=58, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697., source=asf902.gq1.ygridcore.net,53897,1498667454037, destination=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:27,449 INFO [ProcExecWrkr-14] procedure.MasterProcedureScheduler(822): pid=58, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697., source=asf902.gq1.ygridcore.net,53897,1498667454037, destination=asf902.gq1.ygridcore.net,58993,1498667454123 testReplicaGetWithPrimaryAndMetaDown testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,449 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1416): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, server=asf902.gq1.ygridcore.net,53897,1498667454037}] 2017-06-28 16:31:27,453 INFO [ProcExecWrkr-10] procedure.MasterProcedureScheduler(822): pid=59, ppid=58, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, server=asf902.gq1.ygridcore.net,53897,1498667454037 testReplicaGetWithPrimaryAndMetaDown testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,453 INFO [ProcExecWrkr-10] assignment.RegionStateStore(186): pid=59 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697., regionState=CLOSING, regionLocation=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:27,457 DEBUG [regionserver/asf902.gq1.ygridcore.net/67.195.81.146:0.logRoller] regionserver.LogRoller(164): WAL roll requested 2017-06-28 16:31:27,458 INFO [ProcExecWrkr-10] assignment.RegionTransitionProcedure(179): Dispatch pid=59, ppid=58, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, server=asf902.gq1.ygridcore.net,53897,1498667454037; rit=CLOSING, location=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:27,478 INFO [IPC Server handler 2 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40636 is added to blk_1073741853_1029{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75b4545b-7081-4bb7-8c67-5807d44e67e0:NORMAL:127.0.0.1:43805|RBW], ReplicaUC[[DISK]DS-758fa1ec-cf5f-4f33-8dd8-a9ce20d4953e:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-94f4b1e2-c0c4-46fb-8b5d-f3d3d203b341:NORMAL:127.0.0.1:36908|RBW]]} size 83 2017-06-28 16:31:27,479 INFO [IPC Server handler 6 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43805 is added to blk_1073741853_1029{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75b4545b-7081-4bb7-8c67-5807d44e67e0:NORMAL:127.0.0.1:43805|RBW], ReplicaUC[[DISK]DS-758fa1ec-cf5f-4f33-8dd8-a9ce20d4953e:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-94f4b1e2-c0c4-46fb-8b5d-f3d3d203b341:NORMAL:127.0.0.1:36908|RBW]]} size 83 2017-06-28 16:31:27,479 INFO [IPC Server handler 1 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36908 is added to blk_1073741853_1029 size 1003 2017-06-28 16:31:27,480 INFO [regionserver/asf902.gq1.ygridcore.net/67.195.81.146:0.logRoller] wal.AbstractFSWAL(688): Rolled WAL /user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/WALs/asf902.gq1.ygridcore.net,58993,1498667454123/asf902.gq1.ygridcore.net%2C58993%2C1498667454123.1498667481552 with entries=2, filesize=995 B; new WAL /user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/WALs/asf902.gq1.ygridcore.net,58993,1498667454123/asf902.gq1.ygridcore.net%2C58993%2C1498667454123.1498667487457 2017-06-28 16:31:27,480 DEBUG [regionserver/asf902.gq1.ygridcore.net/67.195.81.146:0.logRoller] wal.AbstractFSWAL(757): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40636,DS-758fa1ec-cf5f-4f33-8dd8-a9ce20d4953e,DISK], DatanodeInfoWithStorage[127.0.0.1:36908,DS-94f4b1e2-c0c4-46fb-8b5d-f3d3d203b341,DISK], DatanodeInfoWithStorage[127.0.0.1:43805,DS-75b4545b-7081-4bb7-8c67-5807d44e67e0,DISK]] 2017-06-28 16:31:27,480 INFO [regionserver/asf902.gq1.ygridcore.net/67.195.81.146:0.logRoller] wal.AbstractFSWAL(649): Archiving hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/WALs/asf902.gq1.ygridcore.net,58993,1498667454123/asf902.gq1.ygridcore.net%2C58993%2C1498667454123.1498667481552 to hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/oldWALs/asf902.gq1.ygridcore.net%2C58993%2C1498667454123.1498667481552 2017-06-28 16:31:27,504 INFO [IPC Server handler 2 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36908 is added to blk_1073741854_1030{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-518f10dc-7009-49cd-8bb9-88e593430f4f:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-db41de19-abcb-4f8b-9965-8094f7de5347:NORMAL:127.0.0.1:36908|RBW], ReplicaUC[[DISK]DS-aaac77f0-5ffc-458f-8598-c9877d325015:NORMAL:127.0.0.1:43805|RBW]]} size 83 2017-06-28 16:31:27,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40948] procedure.ProcedureSyncWait(192): waitFor pid=58 2017-06-28 16:31:27,504 INFO [IPC Server handler 6 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43805 is added to blk_1073741854_1030{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-518f10dc-7009-49cd-8bb9-88e593430f4f:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-db41de19-abcb-4f8b-9965-8094f7de5347:NORMAL:127.0.0.1:36908|RBW], ReplicaUC[[DISK]DS-aaac77f0-5ffc-458f-8598-c9877d325015:NORMAL:127.0.0.1:43805|RBW]]} size 83 2017-06-28 16:31:27,505 INFO [IPC Server handler 3 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40636 is added to blk_1073741854_1030{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-518f10dc-7009-49cd-8bb9-88e593430f4f:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-db41de19-abcb-4f8b-9965-8094f7de5347:NORMAL:127.0.0.1:36908|RBW], ReplicaUC[[DISK]DS-aaac77f0-5ffc-458f-8598-c9877d325015:NORMAL:127.0.0.1:43805|RBW]]} size 83 2017-06-28 16:31:27,506 INFO [regionserver/asf902.gq1.ygridcore.net/67.195.81.146:0.logRoller] wal.AbstractFSWAL(688): Rolled WAL /user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/WALs/asf902.gq1.ygridcore.net,58993,1498667454123/asf902.gq1.ygridcore.net%2C58993%2C1498667454123.meta.1498667481585.meta with entries=35, filesize=13.31 KB; new WAL /user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/WALs/asf902.gq1.ygridcore.net,58993,1498667454123/asf902.gq1.ygridcore.net%2C58993%2C1498667454123.meta.1498667487482.meta 2017-06-28 16:31:27,506 DEBUG [regionserver/asf902.gq1.ygridcore.net/67.195.81.146:0.logRoller] wal.AbstractFSWAL(757): Create new FSHLog writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43805,DS-aaac77f0-5ffc-458f-8598-c9877d325015,DISK], DatanodeInfoWithStorage[127.0.0.1:40636,DS-758fa1ec-cf5f-4f33-8dd8-a9ce20d4953e,DISK], DatanodeInfoWithStorage[127.0.0.1:36908,DS-db41de19-abcb-4f8b-9965-8094f7de5347,DISK]] 2017-06-28 16:31:27,506 INFO [regionserver/asf902.gq1.ygridcore.net/67.195.81.146:0.logRoller] wal.AbstractFSWAL(598): Too many WALs; count=3, max=2; forcing flush of 1 regions(s): 1588230740 2017-06-28 16:31:27,511 INFO [MemStoreFlusher.0] regionserver.HRegion(2553): Flushing 5/5 column families, memstore=37.04 KB 2017-06-28 16:31:27,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(175): FLUSHING TO DISK: region hbase:meta,,1.1588230740store: info 2017-06-28 16:31:27,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-06-28 16:31:27,514 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(175): FLUSHING TO DISK: region hbase:meta,,1.1588230740store: rep_barrier 2017-06-28 16:31:27,514 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:0, and the number of cells in new segment is:0 2017-06-28 16:31:27,514 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(175): FLUSHING TO DISK: region hbase:meta,,1.1588230740store: rep_meta 2017-06-28 16:31:27,514 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:0, and the number of cells in new segment is:0 2017-06-28 16:31:27,514 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(175): FLUSHING TO DISK: region hbase:meta,,1.1588230740store: rep_position 2017-06-28 16:31:27,514 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:0, and the number of cells in new segment is:0 2017-06-28 16:31:27,514 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(175): FLUSHING TO DISK: region hbase:meta,,1.1588230740store: table 2017-06-28 16:31:27,514 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-06-28 16:31:27,541 INFO [IPC Server handler 8 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36908 is added to blk_1073741862_1038{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75b4545b-7081-4bb7-8c67-5807d44e67e0:NORMAL:127.0.0.1:43805|RBW], ReplicaUC[[DISK]DS-518f10dc-7009-49cd-8bb9-88e593430f4f:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-db41de19-abcb-4f8b-9965-8094f7de5347:NORMAL:127.0.0.1:36908|FINALIZED]]} size 0 2017-06-28 16:31:27,541 INFO [IPC Server handler 2 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40636 is added to blk_1073741862_1038{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75b4545b-7081-4bb7-8c67-5807d44e67e0:NORMAL:127.0.0.1:43805|RBW], ReplicaUC[[DISK]DS-518f10dc-7009-49cd-8bb9-88e593430f4f:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-db41de19-abcb-4f8b-9965-8094f7de5347:NORMAL:127.0.0.1:36908|FINALIZED]]} size 0 2017-06-28 16:31:27,542 INFO [IPC Server handler 6 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43805 is added to blk_1073741862_1038{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-518f10dc-7009-49cd-8bb9-88e593430f4f:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-db41de19-abcb-4f8b-9965-8094f7de5347:NORMAL:127.0.0.1:36908|FINALIZED], ReplicaUC[[DISK]DS-aaac77f0-5ffc-458f-8598-c9877d325015:NORMAL:127.0.0.1:43805|FINALIZED]]} size 0 2017-06-28 16:31:27,543 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=115, memsize=35.4 K, hasBloomFilter=false, into tmp file hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/hbase/meta/1588230740/.tmp/info/5f88fa7ae62947739303763704839896 2017-06-28 16:31:27,552 INFO [MemStoreFlusher.0] regionserver.StoreFileReader(490): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5f88fa7ae62947739303763704839896 2017-06-28 16:31:27,571 INFO [IPC Server handler 5 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43805 is added to blk_1073741863_1039{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-758fa1ec-cf5f-4f33-8dd8-a9ce20d4953e:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-db41de19-abcb-4f8b-9965-8094f7de5347:NORMAL:127.0.0.1:36908|RBW], ReplicaUC[[DISK]DS-75b4545b-7081-4bb7-8c67-5807d44e67e0:NORMAL:127.0.0.1:43805|FINALIZED]]} size 0 2017-06-28 16:31:27,571 INFO [IPC Server handler 7 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36908 is added to blk_1073741863_1039{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-758fa1ec-cf5f-4f33-8dd8-a9ce20d4953e:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-75b4545b-7081-4bb7-8c67-5807d44e67e0:NORMAL:127.0.0.1:43805|FINALIZED], ReplicaUC[[DISK]DS-94f4b1e2-c0c4-46fb-8b5d-f3d3d203b341:NORMAL:127.0.0.1:36908|FINALIZED]]} size 0 2017-06-28 16:31:27,572 INFO [IPC Server handler 5 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40636 is added to blk_1073741863_1039{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-758fa1ec-cf5f-4f33-8dd8-a9ce20d4953e:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-75b4545b-7081-4bb7-8c67-5807d44e67e0:NORMAL:127.0.0.1:43805|FINALIZED], ReplicaUC[[DISK]DS-94f4b1e2-c0c4-46fb-8b5d-f3d3d203b341:NORMAL:127.0.0.1:36908|FINALIZED]]} size 0 2017-06-28 16:31:27,573 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=115, memsize=1.7 K, hasBloomFilter=false, into tmp file hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/hbase/meta/1588230740/.tmp/table/4bb526774eb240a4acd382ed0fdbadce 2017-06-28 16:31:27,584 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/hbase/meta/1588230740/.tmp/info/5f88fa7ae62947739303763704839896 as hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/hbase/meta/1588230740/info/5f88fa7ae62947739303763704839896 2017-06-28 16:31:27,594 INFO [MemStoreFlusher.0] regionserver.StoreFileReader(490): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5f88fa7ae62947739303763704839896 2017-06-28 16:31:27,595 INFO [MemStoreFlusher.0] regionserver.HStore(1025): Added hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/hbase/meta/1588230740/info/5f88fa7ae62947739303763704839896, entries=57, sequenceid=115, filesize=11.6 K 2017-06-28 16:31:27,598 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/hbase/meta/1588230740/.tmp/table/4bb526774eb240a4acd382ed0fdbadce as hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/hbase/meta/1588230740/table/4bb526774eb240a4acd382ed0fdbadce 2017-06-28 16:31:27,607 INFO [MemStoreFlusher.0] regionserver.HStore(1025): Added hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/hbase/meta/1588230740/table/4bb526774eb240a4acd382ed0fdbadce, entries=9, sequenceid=115, filesize=5.1 K 2017-06-28 16:31:27,608 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=asf902.gq1.ygridcore.net,53897,1498667454037 version=3145728 2017-06-28 16:31:27,610 INFO [MemStoreFlusher.0] regionserver.HRegion(2752): Finished memstore flush of ~37.04 KB/37929, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 99ms, sequenceid=115, compaction requested=false 2017-06-28 16:31:27,610 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=53897] regionserver.RSRpcServices(1519): Close 1cf058a284bbd8c68aa7536052bfa697 without moving 2017-06-28 16:31:27,610 DEBUG [RS_CLOSE_REGION-asf902:53897-1] handler.CloseRegionHandler(90): Processing close of testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,611 DEBUG [RS_CLOSE_REGION-asf902:53897-1] regionserver.HRegion(1573): Closing testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.: disabling compactions & flushes 2017-06-28 16:31:27,611 DEBUG [RS_CLOSE_REGION-asf902:53897-1] regionserver.HRegion(1612): Updates disabled for region testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,612 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.-1] regionserver.HStore(929): Closed colfamily11 2017-06-28 16:31:27,612 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.-1] regionserver.HStore(929): Closed colfamily21 2017-06-28 16:31:27,612 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.-1] regionserver.HStore(929): Closed colfamily31 2017-06-28 16:31:27,612 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.-1] regionserver.HStore(929): Closed info 2017-06-28 16:31:27,620 DEBUG [RS_CLOSE_REGION-asf902:53897-1] wal.WALSplitter(736): Wrote file=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/recovered.edits/4.seqid, newSeqId=4, maxSeqId=2 2017-06-28 16:31:27,621 DEBUG [RS_CLOSE_REGION-asf902:53897-1] coprocessor.CoprocessorHost(290): Stop coprocessor org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro 2017-06-28 16:31:27,621 INFO [RS_CLOSE_REGION-asf902:53897-1] regionserver.HRegion(1725): Closed testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] assignment.RegionTransitionProcedure(200): Received report CLOSED seqId=-1, pid=59, ppid=58, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, server=asf902.gq1.ygridcore.net,53897,1498667454037; rit=CLOSING, location=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:27,623 INFO [ProcExecWrkr-9] assignment.RegionStateStore(186): pid=59 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697., regionState=CLOSED 2017-06-28 16:31:27,623 DEBUG [RS_CLOSE_REGION-asf902:53897-1] handler.CloseRegionHandler(122): Closed testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,629 INFO [ProcExecWrkr-9] procedure2.ProcedureExecutor(1530): Finish suprocedure pid=59, ppid=58, state=SUCCESS; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, server=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:27,631 INFO [ProcExecWrkr-9] procedure2.ProcedureExecutor(1536): Finished subprocedure(s) of pid=58, state=RUNNABLE:MOVE_REGION_ASSIGN; MoveRegionProcedure hri=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697., source=asf902.gq1.ygridcore.net,53897,1498667454037, destination=asf902.gq1.ygridcore.net,58993,1498667454123; resume parent processing. 2017-06-28 16:31:27,631 INFO [ProcExecWrkr-9] procedure2.ProcedureExecutor(1167): Finished pid=59, ppid=58, state=SUCCESS; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, server=asf902.gq1.ygridcore.net,53897,1498667454037 in 178msec 2017-06-28 16:31:27,637 INFO [ProcExecWrkr-15] procedure2.ProcedureExecutor(1416): Initialized subprocedures=[{pid=60, ppid=58, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, target=asf902.gq1.ygridcore.net,58993,1498667454123}] 2017-06-28 16:31:27,640 INFO [ProcExecWrkr-15] procedure.MasterProcedureScheduler(822): pid=60, ppid=58, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, target=asf902.gq1.ygridcore.net,58993,1498667454123 testReplicaGetWithPrimaryAndMetaDown testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,640 INFO [ProcExecWrkr-15] assignment.AssignProcedure(192): Start pid=60, ppid=58, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, target=asf902.gq1.ygridcore.net,58993,1498667454123; rit=OFFLINE, location=asf902.gq1.ygridcore.net,58993,1498667454123; forceNewPlan=false, retain=false 2017-06-28 16:31:27,800 INFO [AssignmentThread] balancer.BaseLoadBalancer(1622): Reassigned 1 regions. 1 retained the pre-restart assignment. 2017-06-28 16:31:27,801 INFO [ProcExecWrkr-11] assignment.RegionStateStore(186): pid=60 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697., regionState=OPENING, regionLocation=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:27,805 INFO [ProcExecWrkr-11] assignment.RegionTransitionProcedure(179): Dispatch pid=60, ppid=58, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, target=asf902.gq1.ygridcore.net,58993,1498667454123; rit=OPENING, location=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:27,956 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=asf902.gq1.ygridcore.net,58993,1498667454123 version=3145728 2017-06-28 16:31:27,957 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=58993] regionserver.RSRpcServices(1899): Open testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,963 DEBUG [RS_OPEN_REGION-asf902:58993-0] regionserver.HRegion(6713): Opening region: {ENCODED => 1cf058a284bbd8c68aa7536052bfa697, NAME => 'testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.', STARTKEY => '', ENDKEY => ''} 2017-06-28 16:31:27,963 INFO [RS_OPEN_REGION-asf902:58993-0] coprocessor.CoprocessorHost(160): System coprocessor org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro was loaded successfully with priority (536870911). 2017-06-28 16:31:27,964 DEBUG [RS_OPEN_REGION-asf902:58993-0] regionserver.MetricsRegionSourceImpl(74): Creating new MetricsRegionSourceImpl for table testReplicaGetWithPrimaryAndMetaDown 1cf058a284bbd8c68aa7536052bfa697 2017-06-28 16:31:27,964 DEBUG [RS_OPEN_REGION-asf902:58993-0] regionserver.HRegion(796): Instantiated testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,967 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily11 2017-06-28 16:31:27,967 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily11 2017-06-28 16:31:27,968 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,968 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,968 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] hfile.CacheConfig(258): Created cacheConfig for colfamily11: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,968 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,972 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily21 2017-06-28 16:31:27,972 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily21 2017-06-28 16:31:27,972 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,972 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,972 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] hfile.CacheConfig(258): Created cacheConfig for colfamily21: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,972 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,975 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily31 2017-06-28 16:31:27,975 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily31 2017-06-28 16:31:27,976 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,976 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,976 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] hfile.CacheConfig(258): Created cacheConfig for colfamily31: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,976 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,979 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info 2017-06-28 16:31:27,979 DEBUG [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info 2017-06-28 16:31:27,979 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:27,979 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:27,979 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] hfile.CacheConfig(258): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:27,980 INFO [StoreOpener-1cf058a284bbd8c68aa7536052bfa697-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:27,981 INFO [RS_OPEN_REGION-asf902:58993-0] regionserver.HRegion(1032): Setting FlushNonSloppyStoresFirstPolicy for the region=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,983 DEBUG [RS_OPEN_REGION-asf902:58993-0] regionserver.HRegion(4147): Found 0 recovered edits file(s) under hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697 2017-06-28 16:31:27,986 DEBUG [RS_OPEN_REGION-asf902:58993-0] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table testReplicaGetWithPrimaryAndMetaDown, use config (33554432) instead 2017-06-28 16:31:27,990 DEBUG [RS_OPEN_REGION-asf902:58993-0] wal.WALSplitter(736): Wrote file=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/recovered.edits/5.seqid, newSeqId=5, maxSeqId=4 2017-06-28 16:31:27,991 INFO [RS_OPEN_REGION-asf902:58993-0] regionserver.HRegion(959): Onlined 1cf058a284bbd8c68aa7536052bfa697; next sequenceid=5 2017-06-28 16:31:27,996 INFO [PostOpenDeployTasks:1cf058a284bbd8c68aa7536052bfa697] regionserver.HRegionServer(2137): Post open deploy tasks for testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40948] assignment.RegionTransitionProcedure(200): Received report OPENED seqId=5, pid=60, ppid=58, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, target=asf902.gq1.ygridcore.net,58993,1498667454123; rit=OPENING, location=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:27,998 DEBUG [PostOpenDeployTasks:1cf058a284bbd8c68aa7536052bfa697] regionserver.HRegionServer(2164): Finished post open deploy task for testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:27,998 INFO [ProcExecWrkr-16] assignment.RegionStateStore(186): pid=60 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697., regionState=OPEN, openSeqNum=5, regionLocation=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:27,999 DEBUG [RS_OPEN_REGION-asf902:58993-0] handler.OpenRegionHandler(126): Opened testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. on asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:28,005 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1530): Finish suprocedure pid=60, ppid=58, state=SUCCESS; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, target=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:28,007 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1536): Finished subprocedure(s) of pid=58, state=RUNNABLE; MoveRegionProcedure hri=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697., source=asf902.gq1.ygridcore.net,53897,1498667454037, destination=asf902.gq1.ygridcore.net,58993,1498667454123; resume parent processing. 2017-06-28 16:31:28,008 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1167): Finished pid=60, ppid=58, state=SUCCESS; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, target=asf902.gq1.ygridcore.net,58993,1498667454123 in 372msec 2017-06-28 16:31:28,009 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1167): Finished pid=58, state=SUCCESS; MoveRegionProcedure hri=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697., source=asf902.gq1.ygridcore.net,53897,1498667454037, destination=asf902.gq1.ygridcore.net,58993,1498667454123 in 670msec 2017-06-28 16:31:28,061 DEBUG [main] ipc.NettyRpcConnection(254): Connecting to asf902.gq1.ygridcore.net/67.195.81.146:58993 2017-06-28 16:31:28,069 INFO [Reader=0,bindAddress=asf902.gq1.ygridcore.net,port=58993] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2017-06-28 16:31:28,072 INFO [Reader=0,bindAddress=asf902.gq1.ygridcore.net,port=58993] ipc.ServerRpcConnection(558): Connection from 67.195.81.146 port: 40335 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf902.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "8bb4d15ba11089879711fe089a985c60df95f6f1" user: "jenkins" date: "Wed Jun 28 16:28:18 UTC 2017" src_checksum: "d448fb06cc9403ac5ae91a7263e0b2fa" version_major: 3 version_minor: 0 2017-06-28 16:31:28,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=53897] regionserver.RSRpcServices(2021): Warming up Region testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:28,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=53897] regionserver.HRegion(6780): HRegion.Warming up region: {ENCODED => da78e90b99801a7d916ba59be21ad54b, NAME => 'testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1} 2017-06-28 16:31:28,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=53897] regionserver.HRegion(796): Instantiated testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:28,086 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily11 2017-06-28 16:31:28,086 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily11 2017-06-28 16:31:28,087 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:28,087 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:28,087 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] hfile.CacheConfig(258): Created cacheConfig for colfamily11: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:28,088 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:28,090 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily21 2017-06-28 16:31:28,090 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily21 2017-06-28 16:31:28,091 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:28,091 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:28,091 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] hfile.CacheConfig(258): Created cacheConfig for colfamily21: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:28,091 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:28,093 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily31 2017-06-28 16:31:28,094 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily31 2017-06-28 16:31:28,094 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:28,094 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:28,094 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] hfile.CacheConfig(258): Created cacheConfig for colfamily31: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:28,094 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:28,097 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info 2017-06-28 16:31:28,097 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info 2017-06-28 16:31:28,098 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:28,098 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:28,098 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] hfile.CacheConfig(258): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:28,098 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:28,099 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=53897] regionserver.HRegion(1032): Setting FlushNonSloppyStoresFirstPolicy for the region=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:28,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40948] master.HMaster(1724): Client=jenkins//67.195.81.146 move hri=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., source=asf902.gq1.ygridcore.net,58993,1498667454123, destination=asf902.gq1.ygridcore.net,53897,1498667454037, running balancer 2017-06-28 16:31:28,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40948] procedure2.ProcedureExecutor(792): Stored pid=61, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., source=asf902.gq1.ygridcore.net,58993,1498667454123, destination=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:28,205 INFO [ProcExecWrkr-7] procedure.MasterProcedureScheduler(822): pid=61, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., source=asf902.gq1.ygridcore.net,58993,1498667454123, destination=asf902.gq1.ygridcore.net,53897,1498667454037 testReplicaGetWithPrimaryAndMetaDown testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:28,205 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1416): Initialized subprocedures=[{pid=62, ppid=61, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, server=asf902.gq1.ygridcore.net,58993,1498667454123}] 2017-06-28 16:31:28,208 INFO [ProcExecWrkr-7] procedure.MasterProcedureScheduler(822): pid=62, ppid=61, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, server=asf902.gq1.ygridcore.net,58993,1498667454123 testReplicaGetWithPrimaryAndMetaDown testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:28,208 INFO [ProcExecWrkr-7] assignment.RegionStateStore(186): pid=62 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., regionState=CLOSING, regionLocation=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:28,212 INFO [ProcExecWrkr-7] assignment.RegionTransitionProcedure(179): Dispatch pid=62, ppid=61, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, server=asf902.gq1.ygridcore.net,58993,1498667454123; rit=CLOSING, location=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:28,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40948] procedure.ProcedureSyncWait(192): waitFor pid=61 2017-06-28 16:31:28,363 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=asf902.gq1.ygridcore.net,58993,1498667454123 version=3145728 2017-06-28 16:31:28,364 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=58993] regionserver.RSRpcServices(1519): Close da78e90b99801a7d916ba59be21ad54b without moving 2017-06-28 16:31:28,364 DEBUG [RS_CLOSE_REGION-asf902:58993-2] handler.CloseRegionHandler(90): Processing close of testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:28,366 DEBUG [RS_CLOSE_REGION-asf902:58993-2] regionserver.HRegion(1573): Closing testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.: disabling compactions & flushes 2017-06-28 16:31:28,366 DEBUG [RS_CLOSE_REGION-asf902:58993-2] regionserver.HRegion(1612): Updates disabled for region testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:28,367 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.-1] regionserver.HStore(929): Closed colfamily11 2017-06-28 16:31:28,367 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.-1] regionserver.HStore(929): Closed colfamily21 2017-06-28 16:31:28,367 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.-1] regionserver.HStore(929): Closed colfamily31 2017-06-28 16:31:28,367 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.-1] regionserver.HStore(929): Closed info 2017-06-28 16:31:28,367 DEBUG [RS_CLOSE_REGION-asf902:58993-2] coprocessor.CoprocessorHost(290): Stop coprocessor org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro 2017-06-28 16:31:28,368 INFO [RS_CLOSE_REGION-asf902:58993-2] regionserver.HRegion(1725): Closed testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:28,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40948] assignment.RegionTransitionProcedure(200): Received report CLOSED seqId=-1, pid=62, ppid=61, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, server=asf902.gq1.ygridcore.net,58993,1498667454123; rit=CLOSING, location=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:28,370 DEBUG [RS_CLOSE_REGION-asf902:58993-2] handler.CloseRegionHandler(122): Closed testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:28,370 INFO [ProcExecWrkr-6] assignment.RegionStateStore(186): pid=62 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., regionState=CLOSED 2017-06-28 16:31:28,376 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1530): Finish suprocedure pid=62, ppid=61, state=SUCCESS; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, server=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:28,378 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1536): Finished subprocedure(s) of pid=61, state=RUNNABLE:MOVE_REGION_ASSIGN; MoveRegionProcedure hri=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., source=asf902.gq1.ygridcore.net,58993,1498667454123, destination=asf902.gq1.ygridcore.net,53897,1498667454037; resume parent processing. 2017-06-28 16:31:28,378 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1167): Finished pid=62, ppid=61, state=SUCCESS; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, server=asf902.gq1.ygridcore.net,58993,1498667454123 in 169msec 2017-06-28 16:31:28,378 INFO [ProcExecWrkr-3] procedure2.ProcedureExecutor(1416): Initialized subprocedures=[{pid=63, ppid=61, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, target=asf902.gq1.ygridcore.net,53897,1498667454037}] 2017-06-28 16:31:28,381 INFO [ProcExecWrkr-3] procedure.MasterProcedureScheduler(822): pid=63, ppid=61, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, target=asf902.gq1.ygridcore.net,53897,1498667454037 testReplicaGetWithPrimaryAndMetaDown testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:28,381 INFO [ProcExecWrkr-3] assignment.AssignProcedure(192): Start pid=63, ppid=61, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, target=asf902.gq1.ygridcore.net,53897,1498667454037; rit=OFFLINE, location=asf902.gq1.ygridcore.net,53897,1498667454037; forceNewPlan=false, retain=false 2017-06-28 16:31:28,438 INFO [asf902.gq1.ygridcore.net,58993,1498667454123_Chore_1] regionserver.HStore(637): Refreshing store files for region hbase:meta,,1_0001.534574363 files to add: [hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/hbase/meta/1588230740/info/5f88fa7ae62947739303763704839896] files to remove: [] 2017-06-28 16:31:28,447 INFO [StoreFileOpenerThread-info-1] regionserver.StoreFileReader(490): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5f88fa7ae62947739303763704839896 2017-06-28 16:31:28,447 DEBUG [asf902.gq1.ygridcore.net,58993,1498667454123_Chore_1] regionserver.HStore(558): loaded hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/hbase/meta/1588230740/info/5f88fa7ae62947739303763704839896, isReference=false, isBulkLoadResult=false, seqid=115, majorCompaction=false 2017-06-28 16:31:28,451 INFO [asf902.gq1.ygridcore.net,58993,1498667454123_Chore_1] regionserver.HStore(637): Refreshing store files for region hbase:meta,,1_0001.534574363 files to add: [hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/hbase/meta/1588230740/table/4bb526774eb240a4acd382ed0fdbadce] files to remove: [] 2017-06-28 16:31:28,459 DEBUG [asf902.gq1.ygridcore.net,58993,1498667454123_Chore_1] regionserver.HStore(558): loaded hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/hbase/meta/1588230740/table/4bb526774eb240a4acd382ed0fdbadce, isReference=false, isBulkLoadResult=false, seqid=115, majorCompaction=false 2017-06-28 16:31:28,536 INFO [AssignmentThread] balancer.BaseLoadBalancer(1622): Reassigned 1 regions. 1 retained the pre-restart assignment. 2017-06-28 16:31:28,536 INFO [ProcExecWrkr-13] assignment.RegionStateStore(186): pid=63 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., regionState=OPENING, regionLocation=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:28,541 INFO [ProcExecWrkr-13] assignment.RegionTransitionProcedure(179): Dispatch pid=63, ppid=61, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, target=asf902.gq1.ygridcore.net,53897,1498667454037; rit=OPENING, location=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:28,691 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=asf902.gq1.ygridcore.net,53897,1498667454037 version=3145728 2017-06-28 16:31:28,693 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=53897] regionserver.RSRpcServices(1899): Open testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:28,698 DEBUG [RS_OPEN_REGION-asf902:53897-2] regionserver.HRegion(6713): Opening region: {ENCODED => da78e90b99801a7d916ba59be21ad54b, NAME => 'testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1} 2017-06-28 16:31:28,699 INFO [RS_OPEN_REGION-asf902:53897-2] coprocessor.CoprocessorHost(160): System coprocessor org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro was loaded successfully with priority (536870911). 2017-06-28 16:31:28,699 DEBUG [RS_OPEN_REGION-asf902:53897-2] regionserver.MetricsRegionSourceImpl(74): Creating new MetricsRegionSourceImpl for table testReplicaGetWithPrimaryAndMetaDown da78e90b99801a7d916ba59be21ad54b 2017-06-28 16:31:28,699 DEBUG [RS_OPEN_REGION-asf902:53897-2] regionserver.HRegion(796): Instantiated testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:28,703 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily11 2017-06-28 16:31:28,703 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily11 2017-06-28 16:31:28,704 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:28,704 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:28,704 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] hfile.CacheConfig(258): Created cacheConfig for colfamily11: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:28,705 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:28,708 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily21 2017-06-28 16:31:28,708 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily21 2017-06-28 16:31:28,708 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:28,708 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:28,708 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] hfile.CacheConfig(258): Created cacheConfig for colfamily21: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:28,709 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:28,711 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily31 2017-06-28 16:31:28,711 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily31 2017-06-28 16:31:28,711 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:28,711 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:28,711 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] hfile.CacheConfig(258): Created cacheConfig for colfamily31: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:28,712 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:28,714 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info 2017-06-28 16:31:28,714 DEBUG [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] util.FSUtils(243): Set storagePolicy=HOT for path=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info 2017-06-28 16:31:28,714 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.CompactingMemStore(118): Setting in-memory flush size threshold to 8388608 2017-06-28 16:31:28,714 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] regionserver.HStore(279): Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore 2017-06-28 16:31:28,715 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] hfile.CacheConfig(258): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=4, currentSize=769152, freeSize=1043193152, maxSize=1043962304, heapSize=769152, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2017-06-28 16:31:28,715 INFO [StoreOpener-da78e90b99801a7d916ba59be21ad54b-1] compactions.CompactionConfiguration(145): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2017-06-28 16:31:28,716 INFO [RS_OPEN_REGION-asf902:53897-2] regionserver.HRegion(1032): Setting FlushNonSloppyStoresFirstPolicy for the region=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:28,716 DEBUG [RS_OPEN_REGION-asf902:53897-2] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table testReplicaGetWithPrimaryAndMetaDown, use config (33554432) instead 2017-06-28 16:31:28,716 INFO [RS_OPEN_REGION-asf902:53897-2] regionserver.HRegion(959): Onlined da78e90b99801a7d916ba59be21ad54b; next sequenceid=2 2017-06-28 16:31:28,716 INFO [PostOpenDeployTasks:da78e90b99801a7d916ba59be21ad54b] regionserver.HRegionServer(2137): Post open deploy tasks for testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:28,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40948] assignment.RegionTransitionProcedure(200): Received report OPENED seqId=2, pid=63, ppid=61, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, target=asf902.gq1.ygridcore.net,53897,1498667454037; rit=OPENING, location=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:28,718 DEBUG [PostOpenDeployTasks:da78e90b99801a7d916ba59be21ad54b] regionserver.HRegionServer(2164): Finished post open deploy task for testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:28,718 INFO [ProcExecWrkr-12] assignment.RegionStateStore(186): pid=63 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., regionState=OPEN, openSeqNum=2, regionLocation=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:28,720 DEBUG [RS_OPEN_REGION-asf902:53897-2] handler.OpenRegionHandler(126): Opened testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. on asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:28,726 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1530): Finish suprocedure pid=63, ppid=61, state=SUCCESS; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, target=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:28,728 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1536): Finished subprocedure(s) of pid=61, state=RUNNABLE; MoveRegionProcedure hri=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., source=asf902.gq1.ygridcore.net,58993,1498667454123, destination=asf902.gq1.ygridcore.net,53897,1498667454037; resume parent processing. 2017-06-28 16:31:28,728 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1167): Finished pid=63, ppid=61, state=SUCCESS; AssignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, target=asf902.gq1.ygridcore.net,53897,1498667454037 in 346msec 2017-06-28 16:31:28,830 INFO [ProcExecWrkr-8] procedure2.ProcedureExecutor(1167): Finished pid=61, state=SUCCESS; MoveRegionProcedure hri=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., source=asf902.gq1.ygridcore.net,58993,1498667454123, destination=asf902.gq1.ygridcore.net,53897,1498667454037 in 628msec 2017-06-28 16:31:28,857 DEBUG [main] ipc.NettyRpcConnection(254): Connecting to asf902.gq1.ygridcore.net/67.195.81.146:53897 2017-06-28 16:31:28,867 INFO [Reader=1,bindAddress=asf902.gq1.ygridcore.net,port=53897] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2017-06-28 16:31:28,870 INFO [Reader=1,bindAddress=asf902.gq1.ygridcore.net,port=53897] ipc.ServerRpcConnection(558): Connection from 67.195.81.146 port: 36058 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf902.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "8bb4d15ba11089879711fe089a985c60df95f6f1" user: "jenkins" date: "Wed Jun 28 16:28:18 UTC 2017" src_checksum: "d448fb06cc9403ac5ae91a7263e0b2fa" version_major: 3 version_minor: 0 2017-06-28 16:31:28,877 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(215): Scan, We're replica region 0 2017-06-28 16:31:28,890 INFO [main] client.TestReplicaWithCluster(734): meta locations [region=hbase:meta,,1.1588230740, hostname=asf902.gq1.ygridcore.net,58993,1498667454123, seqNum=0, region=hbase:meta,,1_0001.534574363, hostname=asf902.gq1.ygridcore.net,58993,1498667454123, seqNum=0] 2017-06-28 16:31:28,890 INFO [main] client.TestReplicaWithCluster(735): table locations [region=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697., hostname=asf902.gq1.ygridcore.net,58993,1498667454123, seqNum=5, region=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., hostname=asf902.gq1.ygridcore.net,53897,1498667454037, seqNum=2] 2017-06-28 16:31:28,895 DEBUG [main] regionserver.HRegion(2236): NOT flushing memstore for region testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., flushing=false, writesEnabled=false 2017-06-28 16:31:28,896 INFO [main] regionserver.HRegion(2553): Flushing 4/4 column families, memstore=183 B 2017-06-28 16:31:28,897 DEBUG [main] regionserver.CompactingMemStore(175): FLUSHING TO DISK: region testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.store: colfamily11 2017-06-28 16:31:28,897 DEBUG [main] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:0, and the number of cells in new segment is:0 2017-06-28 16:31:28,897 DEBUG [main] regionserver.CompactingMemStore(175): FLUSHING TO DISK: region testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.store: colfamily21 2017-06-28 16:31:28,898 DEBUG [main] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:0, and the number of cells in new segment is:0 2017-06-28 16:31:28,898 DEBUG [main] regionserver.CompactingMemStore(175): FLUSHING TO DISK: region testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.store: colfamily31 2017-06-28 16:31:28,898 DEBUG [main] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:0, and the number of cells in new segment is:0 2017-06-28 16:31:28,898 DEBUG [main] regionserver.CompactingMemStore(175): FLUSHING TO DISK: region testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.store: info 2017-06-28 16:31:28,898 DEBUG [main] regionserver.CompactionPipeline(133): Swapping pipeline suffix. Just before the swap the number of segments in pipeline is:1, and the number of cells in new segment is:0 2017-06-28 16:31:28,916 INFO [IPC Server handler 7 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43805 is added to blk_1073741864_1040{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-94f4b1e2-c0c4-46fb-8b5d-f3d3d203b341:NORMAL:127.0.0.1:36908|RBW], ReplicaUC[[DISK]DS-518f10dc-7009-49cd-8bb9-88e593430f4f:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-aaac77f0-5ffc-458f-8598-c9877d325015:NORMAL:127.0.0.1:43805|RBW]]} size 0 2017-06-28 16:31:28,916 INFO [IPC Server handler 5 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40636 is added to blk_1073741864_1040{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-94f4b1e2-c0c4-46fb-8b5d-f3d3d203b341:NORMAL:127.0.0.1:36908|RBW], ReplicaUC[[DISK]DS-518f10dc-7009-49cd-8bb9-88e593430f4f:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-aaac77f0-5ffc-458f-8598-c9877d325015:NORMAL:127.0.0.1:43805|RBW]]} size 0 2017-06-28 16:31:28,917 INFO [IPC Server handler 4 on 59739] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36908 is added to blk_1073741864_1040{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-518f10dc-7009-49cd-8bb9-88e593430f4f:NORMAL:127.0.0.1:40636|RBW], ReplicaUC[[DISK]DS-aaac77f0-5ffc-458f-8598-c9877d325015:NORMAL:127.0.0.1:43805|RBW], ReplicaUC[[DISK]DS-db41de19-abcb-4f8b-9965-8094f7de5347:NORMAL:127.0.0.1:36908|FINALIZED]]} size 0 2017-06-28 16:31:28,918 INFO [main] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=8, memsize=183, hasBloomFilter=false, into tmp file hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/.tmp/info/e95e209200c0460887a6b8055bf04294 2017-06-28 16:31:28,927 DEBUG [main] regionserver.HRegionFileSystem(464): Committing store file hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/.tmp/info/e95e209200c0460887a6b8055bf04294 as hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info/e95e209200c0460887a6b8055bf04294 2017-06-28 16:31:28,937 INFO [main] regionserver.HStore(1025): Added hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info/e95e209200c0460887a6b8055bf04294, entries=1, sequenceid=8, filesize=4.9 K 2017-06-28 16:31:28,939 INFO [main] regionserver.HRegion(2752): Finished memstore flush of ~183 B/183, currentsize=0 B/0 for region testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. in 43ms, sequenceid=8, compaction requested=false 2017-06-28 16:31:28,948 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@ff684e1] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:43805 to delete [blk_1073741856_1032, blk_1073741857_1033, blk_1073741845_1021, blk_1073741846_1022, blk_1073741851_1027, blk_1073741852_1028] 2017-06-28 16:31:29,365 INFO [asf902.gq1.ygridcore.net,53897,1498667454037_Chore_1] regionserver.HStore(637): Refreshing store files for region testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. files to add: [hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info/e95e209200c0460887a6b8055bf04294] files to remove: [] 2017-06-28 16:31:29,374 DEBUG [asf902.gq1.ygridcore.net,53897,1498667454037_Chore_1] regionserver.HStore(558): loaded hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info/e95e209200c0460887a6b8055bf04294, isReference=false, isBulkLoadResult=false, seqid=8, majorCompaction=false 2017-06-28 16:31:30,786 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2017-06-28 16:31:30,944 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(180): Get, throw Region Server Stopped Exceptoin for region {ENCODED => 1cf058a284bbd8c68aa7536052bfa697, NAME => 'testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.', STARTKEY => '', ENDKEY => ''} 2017-06-28 16:31:30,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=58993] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Get size: 173 connection: 67.195.81.146:40068 deadline: 1498667550943 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preGetOp(TestReplicaWithCluster.java:183) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$24.call(RegionCoprocessorHost.java:815) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1701) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:811) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2492) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2439) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41137) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,060 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(209): Scan, throw Region Server Stopped Exceptoin for replica {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2017-06-28 16:31:31,060 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Scan size: 185 connection: 67.195.81.146:40068 deadline: 1498667551057 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,164 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(209): Scan, throw Region Server Stopped Exceptoin for replica {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2017-06-28 16:31:31,164 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Scan size: 185 connection: 67.195.81.146:40068 deadline: 1498667551057 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,167 DEBUG [hconnection-0x7275c74b-shared-pool15-t50] client.ScannerCallableWithReplicas(200): Scan with primary region returns org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=2, exceptions: Wed Jun 28 16:31:31 UTC 2017, RpcRetryingCaller{globalStartTime=1498667491056, pause=100, maxAttempts=2}, org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) Wed Jun 28 16:31:31 UTC 2017, RpcRetryingCaller{globalStartTime=1498667491056, pause=100, maxAttempts=2}, org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,170 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(218): Scan, We're replica region 1 2017-06-28 16:31:31,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=53897] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Get size: 173 connection: 67.195.81.146:35818 deadline: 1498667551175 org.apache.hadoop.hbase.NotServingRegionException: testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. is not online on asf902.gq1.ygridcore.net,53897,1498667454037 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3268) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1373) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2424) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41137) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,178 DEBUG [main] client.RpcRetryingCallerWithReadReplicas(222): Primary replica returns org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=2, exceptions: Wed Jun 28 16:31:30 UTC 2017, RpcRetryingCaller{globalStartTime=1498667490942, pause=100, maxAttempts=2}, org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preGetOp(TestReplicaWithCluster.java:183) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$24.call(RegionCoprocessorHost.java:815) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1701) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:811) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2492) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2439) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41137) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) Wed Jun 28 16:31:31 UTC 2017, RpcRetryingCaller{globalStartTime=1498667490942, pause=100, maxAttempts=2}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. is not online on asf902.gq1.ygridcore.net,53897,1498667454037 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3268) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1373) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2424) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41137) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,189 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(209): Scan, throw Region Server Stopped Exceptoin for replica {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2017-06-28 16:31:31,189 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Scan size: 185 connection: 67.195.81.146:40068 deadline: 1498667551186 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,293 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(209): Scan, throw Region Server Stopped Exceptoin for replica {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2017-06-28 16:31:31,294 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Scan size: 185 connection: 67.195.81.146:40068 deadline: 1498667551185 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,296 DEBUG [main] client.ScannerCallableWithReplicas(200): Scan with primary region returns org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=2, exceptions: Wed Jun 28 16:31:31 UTC 2017, RpcRetryingCaller{globalStartTime=1498667491185, pause=100, maxAttempts=2}, org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) Wed Jun 28 16:31:31 UTC 2017, RpcRetryingCaller{globalStartTime=1498667491185, pause=100, maxAttempts=2}, org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,300 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(218): Scan, We're replica region 1 2017-06-28 16:31:31,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=53897] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Get size: 173 connection: 67.195.81.146:35818 deadline: 1498667551308 org.apache.hadoop.hbase.NotServingRegionException: testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. is not online on asf902.gq1.ygridcore.net,53897,1498667454037 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3268) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1373) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2424) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41137) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,417 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(209): Scan, throw Region Server Stopped Exceptoin for replica {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2017-06-28 16:31:31,417 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Scan size: 185 connection: 67.195.81.146:40068 deadline: 1498667551413 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,521 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(209): Scan, throw Region Server Stopped Exceptoin for replica {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2017-06-28 16:31:31,522 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Scan size: 185 connection: 67.195.81.146:40068 deadline: 1498667551414 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,526 DEBUG [hconnection-0x7275c74b-shared-pool15-t52] client.ScannerCallableWithReplicas(200): Scan with primary region returns org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=2, exceptions: Wed Jun 28 16:31:31 UTC 2017, RpcRetryingCaller{globalStartTime=1498667491413, pause=100, maxAttempts=2}, org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) Wed Jun 28 16:31:31 UTC 2017, RpcRetryingCaller{globalStartTime=1498667491413, pause=100, maxAttempts=2}, org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,529 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(218): Scan, We're replica region 1 2017-06-28 16:31:31,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=53897] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Get size: 173 connection: 67.195.81.146:35818 deadline: 1498667551582 org.apache.hadoop.hbase.NotServingRegionException: testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. is not online on asf902.gq1.ygridcore.net,53897,1498667454037 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3268) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1373) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2424) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41137) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,585 DEBUG [main] client.RpcRetryingCallerWithReadReplicas(222): Primary replica returns org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=2, exceptions: Wed Jun 28 16:31:31 UTC 2017, RpcRetryingCaller{globalStartTime=1498667491308, pause=100, maxAttempts=2}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. is not online on asf902.gq1.ygridcore.net,53897,1498667454037 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3268) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1373) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2424) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41137) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) Wed Jun 28 16:31:31 UTC 2017, RpcRetryingCaller{globalStartTime=1498667491308, pause=100, maxAttempts=2}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. is not online on asf902.gq1.ygridcore.net,53897,1498667454037 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3268) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1373) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2424) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41137) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=58993] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Get size: 178 connection: 67.195.81.146:40068 deadline: 1498667551586 org.apache.hadoop.hbase.NotServingRegionException: testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. is not online on asf902.gq1.ygridcore.net,58993,1498667454123 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3268) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1373) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2424) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41137) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,590 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2017-06-28 16:31:31,590 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2017-06-28 16:31:31,693 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(209): Scan, throw Region Server Stopped Exceptoin for replica {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2017-06-28 16:31:31,694 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Scan size: 185 connection: 67.195.81.146:40068 deadline: 1498667551690 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,798 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(209): Scan, throw Region Server Stopped Exceptoin for replica {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2017-06-28 16:31:31,798 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Scan size: 185 connection: 67.195.81.146:40068 deadline: 1498667551690 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,800 DEBUG [hconnection-0x7275c74b-shared-pool15-t53] client.ScannerCallableWithReplicas(200): Scan with primary region returns org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=2, exceptions: Wed Jun 28 16:31:31 UTC 2017, RpcRetryingCaller{globalStartTime=1498667491690, pause=100, maxAttempts=2}, org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) Wed Jun 28 16:31:31 UTC 2017, RpcRetryingCaller{globalStartTime=1498667491690, pause=100, maxAttempts=2}, org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server asf902.gq1.ygridcore.net,58993,1498667454123 not running at org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro.preScannerOpen(TestReplicaWithCluster.java:213) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$48.call(RegionCoprocessorHost.java:1266) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1662) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1718) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1261) at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2899) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3205) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41141) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,804 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(218): Scan, We're replica region 1 2017-06-28 16:31:31,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=58993] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Get size: 178 connection: 67.195.81.146:40068 deadline: 1498667551807 org.apache.hadoop.hbase.NotServingRegionException: testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. is not online on asf902.gq1.ygridcore.net,58993,1498667454123 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3268) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1373) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2424) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41137) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,810 DEBUG [main] client.ResultBoundedCompletionService(226): Replica 1 returns org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=2, exceptions: Wed Jun 28 16:31:31 UTC 2017, RpcRetryingCaller{globalStartTime=1498667491586, pause=100, maxAttempts=2}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. is not online on asf902.gq1.ygridcore.net,58993,1498667454123 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3268) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1373) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2424) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41137) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) Wed Jun 28 16:31:31 UTC 2017, RpcRetryingCaller{globalStartTime=1498667491586, pause=100, maxAttempts=2}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. is not online on asf902.gq1.ygridcore.net,58993,1498667454123 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3268) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1373) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2424) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41137) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:406) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258) 2017-06-28 16:31:31,813 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(465): master:40948-0x15cef8a87ad0000, quorum=localhost:52259, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2017-06-28 16:31:31,813 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] master.MasterRpcServices(234): Client=jenkins//67.195.81.146 set balanceSwitch=true 2017-06-28 16:31:31,815 INFO [main] client.HBaseAdmin$10(775): Started disable of testReplicaGetWithPrimaryAndMetaDown 2017-06-28 16:31:31,815 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] master.HMaster$12(2348): Client=jenkins//67.195.81.146 disable testReplicaGetWithPrimaryAndMetaDown 2017-06-28 16:31:31,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] procedure2.ProcedureExecutor(792): Stored pid=64, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testReplicaGetWithPrimaryAndMetaDown 2017-06-28 16:31:31,923 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(186): Get, We're replica region 0 2017-06-28 16:31:31,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] master.MasterRpcServices(1011): Checking to see if procedure is done procId=64 2017-06-28 16:31:31,925 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(186): Get, We're replica region 0 2017-06-28 16:31:31,930 DEBUG [ProcExecWrkr-14] hbase.MetaTableAccessor(1391): Put{"totalColumns":1,"row":"testReplicaGetWithPrimaryAndMetaDown","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1498667491930}]}} 2017-06-28 16:31:31,935 INFO [ProcExecWrkr-14] hbase.MetaTableAccessor(1767): Updated table testReplicaGetWithPrimaryAndMetaDown state to DISABLING in META 2017-06-28 16:31:31,937 INFO [ProcExecWrkr-14] procedure2.ProcedureExecutor(1416): Initialized subprocedures=[{pid=65, ppid=64, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, server=asf902.gq1.ygridcore.net,58993,1498667454123}, {pid=66, ppid=64, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, server=asf902.gq1.ygridcore.net,53897,1498667454037}] 2017-06-28 16:31:31,941 INFO [ProcExecWrkr-10] procedure.MasterProcedureScheduler(822): pid=66, ppid=64, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, server=asf902.gq1.ygridcore.net,53897,1498667454037 testReplicaGetWithPrimaryAndMetaDown testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:31,941 INFO [ProcExecWrkr-9] procedure.MasterProcedureScheduler(822): pid=65, ppid=64, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, server=asf902.gq1.ygridcore.net,58993,1498667454123 testReplicaGetWithPrimaryAndMetaDown testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:31,941 INFO [ProcExecWrkr-10] assignment.RegionStateStore(186): pid=66 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., regionState=CLOSING, regionLocation=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:31,941 INFO [ProcExecWrkr-9] assignment.RegionStateStore(186): pid=65 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697., regionState=CLOSING, regionLocation=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:31,945 INFO [ProcExecWrkr-10] assignment.RegionTransitionProcedure(179): Dispatch pid=66, ppid=64, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, server=asf902.gq1.ygridcore.net,53897,1498667454037; rit=CLOSING, location=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:31,946 INFO [ProcExecWrkr-9] assignment.RegionTransitionProcedure(179): Dispatch pid=65, ppid=64, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, server=asf902.gq1.ygridcore.net,58993,1498667454123; rit=CLOSING, location=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:31,949 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@ff684e1] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:40636 to delete [blk_1073741856_1032, blk_1073741857_1033, blk_1073741851_1027, blk_1073741852_1028] 2017-06-28 16:31:32,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] master.MasterRpcServices(1011): Checking to see if procedure is done procId=64 2017-06-28 16:31:32,096 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=asf902.gq1.ygridcore.net,53897,1498667454037 version=3145728 2017-06-28 16:31:32,096 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(105): Using procedure batch rpc execution for serverName=asf902.gq1.ygridcore.net,58993,1498667454123 version=3145728 2017-06-28 16:31:32,097 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=53897] regionserver.RSRpcServices(1519): Close da78e90b99801a7d916ba59be21ad54b without moving 2017-06-28 16:31:32,097 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=58993] regionserver.RSRpcServices(1519): Close 1cf058a284bbd8c68aa7536052bfa697 without moving 2017-06-28 16:31:32,097 DEBUG [RS_CLOSE_REGION-asf902:53897-2] handler.CloseRegionHandler(90): Processing close of testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:32,098 DEBUG [RS_CLOSE_REGION-asf902:58993-0] handler.CloseRegionHandler(90): Processing close of testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:32,098 DEBUG [RS_CLOSE_REGION-asf902:53897-2] regionserver.HRegion(1573): Closing testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.: disabling compactions & flushes 2017-06-28 16:31:32,098 DEBUG [RS_CLOSE_REGION-asf902:58993-0] regionserver.HRegion(1573): Closing testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.: disabling compactions & flushes 2017-06-28 16:31:32,098 DEBUG [RS_CLOSE_REGION-asf902:53897-2] regionserver.HRegion(1612): Updates disabled for region testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:32,098 DEBUG [RS_CLOSE_REGION-asf902:58993-0] regionserver.HRegion(1612): Updates disabled for region testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:32,099 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.-1] regionserver.HStore(929): Closed colfamily11 2017-06-28 16:31:32,099 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.-1] regionserver.HStore(929): Closed colfamily21 2017-06-28 16:31:32,099 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.-1] regionserver.HStore(929): Closed colfamily11 2017-06-28 16:31:32,099 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.-1] regionserver.HStore(929): Closed colfamily31 2017-06-28 16:31:32,099 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.-1] regionserver.HStore(929): Closed colfamily21 2017-06-28 16:31:32,100 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.-1] regionserver.HStore(929): Closed colfamily31 2017-06-28 16:31:32,104 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.-1] regionserver.HStore(929): Closed info 2017-06-28 16:31:32,104 INFO [StoreCloserThread-testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.-1] regionserver.HStore(929): Closed info 2017-06-28 16:31:32,105 DEBUG [RS_CLOSE_REGION-asf902:53897-2] coprocessor.CoprocessorHost(290): Stop coprocessor org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro 2017-06-28 16:31:32,105 INFO [RS_CLOSE_REGION-asf902:53897-2] regionserver.HRegion(1725): Closed testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:32,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] assignment.RegionTransitionProcedure(200): Received report CLOSED seqId=-1, pid=66, ppid=64, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, server=asf902.gq1.ygridcore.net,53897,1498667454037; rit=CLOSING, location=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:32,106 DEBUG [RS_CLOSE_REGION-asf902:53897-2] handler.CloseRegionHandler(122): Closed testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. 2017-06-28 16:31:32,106 INFO [ProcExecWrkr-2] assignment.RegionStateStore(186): pid=66 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b., regionState=CLOSED 2017-06-28 16:31:32,112 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1530): Finish suprocedure pid=66, ppid=64, state=SUCCESS; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, server=asf902.gq1.ygridcore.net,53897,1498667454037 2017-06-28 16:31:32,113 INFO [ProcExecWrkr-2] procedure2.ProcedureExecutor(1167): Finished pid=66, ppid=64, state=SUCCESS; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=da78e90b99801a7d916ba59be21ad54b, server=asf902.gq1.ygridcore.net,53897,1498667454037 in 173msec 2017-06-28 16:31:32,114 DEBUG [RS_CLOSE_REGION-asf902:58993-0] wal.WALSplitter(736): Wrote file=hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/recovered.edits/11.seqid, newSeqId=11, maxSeqId=5 2017-06-28 16:31:32,115 DEBUG [RS_CLOSE_REGION-asf902:58993-0] coprocessor.CoprocessorHost(290): Stop coprocessor org.apache.hadoop.hbase.client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro 2017-06-28 16:31:32,115 INFO [RS_CLOSE_REGION-asf902:58993-0] regionserver.HRegion(1725): Closed testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:32,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] assignment.RegionTransitionProcedure(200): Received report CLOSED seqId=-1, pid=65, ppid=64, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, server=asf902.gq1.ygridcore.net,58993,1498667454123; rit=CLOSING, location=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:32,116 INFO [ProcExecWrkr-15] assignment.RegionStateStore(186): pid=65 updating hbase:meta row=testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697., regionState=CLOSED 2017-06-28 16:31:32,117 DEBUG [RS_CLOSE_REGION-asf902:58993-0] handler.CloseRegionHandler(122): Closed testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. 2017-06-28 16:31:32,124 INFO [ProcExecWrkr-15] procedure2.ProcedureExecutor(1530): Finish suprocedure pid=65, ppid=64, state=SUCCESS; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, server=asf902.gq1.ygridcore.net,58993,1498667454123 2017-06-28 16:31:32,126 INFO [ProcExecWrkr-15] procedure2.ProcedureExecutor(1536): Finished subprocedure(s) of pid=64, state=RUNNABLE:DISABLE_TABLE_SET_DISABLED_TABLE_STATE; DisableTableProcedure table=testReplicaGetWithPrimaryAndMetaDown; resume parent processing. 2017-06-28 16:31:32,126 INFO [ProcExecWrkr-15] procedure2.ProcedureExecutor(1167): Finished pid=65, ppid=64, state=SUCCESS; UnassignProcedure table=testReplicaGetWithPrimaryAndMetaDown, region=1cf058a284bbd8c68aa7536052bfa697, server=asf902.gq1.ygridcore.net,58993,1498667454123 in 185msec 2017-06-28 16:31:32,127 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(1391): Put{"totalColumns":1,"row":"testReplicaGetWithPrimaryAndMetaDown","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1498667492126}]}} 2017-06-28 16:31:32,131 INFO [ProcExecWrkr-11] hbase.MetaTableAccessor(1767): Updated table testReplicaGetWithPrimaryAndMetaDown state to DISABLED in META 2017-06-28 16:31:32,131 INFO [ProcExecWrkr-11] procedure.DisableTableProcedure(291): Disabled table, testReplicaGetWithPrimaryAndMetaDown, is completed. 2017-06-28 16:31:32,135 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1167): Finished pid=64, state=SUCCESS; DisableTableProcedure table=testReplicaGetWithPrimaryAndMetaDown in 318msec 2017-06-28 16:31:32,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] master.MasterRpcServices(1011): Checking to see if procedure is done procId=64 2017-06-28 16:31:32,228 INFO [main] client.HBaseAdmin$TableFuture(3584): Operation: DISABLE, Table Name: default:testReplicaGetWithPrimaryAndMetaDown completed 2017-06-28 16:31:32,229 INFO [main] client.HBaseAdmin$10(775): Started disable of testReplicaGetWithPrimaryAndMetaDown 2017-06-28 16:31:32,230 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] master.HMaster$12(2348): Client=jenkins//67.195.81.146 disable testReplicaGetWithPrimaryAndMetaDown 2017-06-28 16:31:32,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] procedure2.ProcedureExecutor(792): Stored pid=67, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testReplicaGetWithPrimaryAndMetaDown 2017-06-28 16:31:32,335 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(186): Get, We're replica region 0 2017-06-28 16:31:32,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] master.MasterRpcServices(1011): Checking to see if procedure is done procId=67 2017-06-28 16:31:32,338 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(186): Get, We're replica region 0 2017-06-28 16:31:32,339 INFO [ProcExecWrkr-16] procedure.DisableTableProcedure(240): Table testReplicaGetWithPrimaryAndMetaDown isn't enabled;is DISABLED; skipping disable 2017-06-28 16:31:32,343 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1261): Rolled back pid=67, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.TableNotEnabledException via master-disable-table:org.apache.hadoop.hbase.TableNotEnabledException: testReplicaGetWithPrimaryAndMetaDown state is DISABLED; DisableTableProcedure table=testReplicaGetWithPrimaryAndMetaDown exec-time=111msec 2017-06-28 16:31:32,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] master.MasterRpcServices(1011): Checking to see if procedure is done procId=67 2017-06-28 16:31:32,440 INFO [main] client.HBaseAdmin$TableFuture(3591): Operation: DISABLE, Table Name: default:testReplicaGetWithPrimaryAndMetaDown failed with testReplicaGetWithPrimaryAndMetaDown state is DISABLED 2017-06-28 16:31:32,440 DEBUG [main] hbase.HBaseTestingUtility(1745): Table: testReplicaGetWithPrimaryAndMetaDown already disabled, so just deleting it. 2017-06-28 16:31:32,441 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] master.HMaster$6(2126): Client=jenkins//67.195.81.146 delete testReplicaGetWithPrimaryAndMetaDown 2017-06-28 16:31:32,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] procedure2.ProcedureExecutor(792): Stored pid=68, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testReplicaGetWithPrimaryAndMetaDown 2017-06-28 16:31:32,547 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(186): Get, We're replica region 0 2017-06-28 16:31:32,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] master.MasterRpcServices(1011): Checking to see if procedure is done procId=68 2017-06-28 16:31:32,549 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(186): Get, We're replica region 0 2017-06-28 16:31:32,550 DEBUG [ProcExecWrkr-1] procedure.DeleteTableProcedure(99): Waiting for 'testReplicaGetWithPrimaryAndMetaDown' regions in transition 2017-06-28 16:31:32,552 DEBUG [ProcExecWrkr-1] procedure.DeleteTableProcedure(110): delete 'testReplicaGetWithPrimaryAndMetaDown' regions from META 2017-06-28 16:31:32,552 DEBUG [ProcExecWrkr-1] hbase.MetaTableAccessor(1442): Delete{"totalColumns":1,"row":"testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1498667492552}]},"ts":9223372036854775807}, Delete{"totalColumns":1,"row":"testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1498667492552}]},"ts":9223372036854775807} 2017-06-28 16:31:32,556 INFO [ProcExecWrkr-1] hbase.MetaTableAccessor(1980): Deleted [{ENCODED => 1cf058a284bbd8c68aa7536052bfa697, NAME => 'testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.', STARTKEY => '', ENDKEY => ''}, {ENCODED => da78e90b99801a7d916ba59be21ad54b, NAME => 'testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1}] 2017-06-28 16:31:32,558 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=58993] client.TestReplicaWithCluster$RegionServerHostingPrimayMetaRegionSlowOrStopCopro(215): Scan, We're replica region 0 2017-06-28 16:31:32,562 DEBUG [ProcExecWrkr-1] procedure.DeleteTableProcedure(115): delete 'testReplicaGetWithPrimaryAndMetaDown' from filesystem 2017-06-28 16:31:32,566 DEBUG [ProcExecWrkr-1] procedure.DeleteTableProcedure(313): Archiving region testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697. from FS 2017-06-28 16:31:32,566 DEBUG [ProcExecWrkr-1] backup.HFileArchiver(114): ARCHIVING hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/.tmp/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697 2017-06-28 16:31:32,572 DEBUG [ProcExecWrkr-1] backup.HFileArchiver(155): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/.tmp/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily11, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/.tmp/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily21, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/.tmp/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/colfamily31, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/.tmp/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/.tmp/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/recovered.edits] 2017-06-28 16:31:32,585 DEBUG [ProcExecWrkr-1] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/.tmp/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info/e95e209200c0460887a6b8055bf04294, to hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/archive/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/info/e95e209200c0460887a6b8055bf04294 2017-06-28 16:31:32,592 DEBUG [ProcExecWrkr-1] backup.HFileArchiver(451): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/.tmp/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/recovered.edits/11.seqid, to hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/archive/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697/recovered.edits/11.seqid 2017-06-28 16:31:32,593 INFO [IPC Server handler 4 on 59739] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741859_1035 127.0.0.1:40636 127.0.0.1:36908 127.0.0.1:43805 2017-06-28 16:31:32,593 DEBUG [ProcExecWrkr-1] backup.HFileArchiver(466): Deleted hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/.tmp/data/default/testReplicaGetWithPrimaryAndMetaDown/1cf058a284bbd8c68aa7536052bfa697 2017-06-28 16:31:32,593 DEBUG [ProcExecWrkr-1] procedure.DeleteTableProcedure(313): Archiving region testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b. from FS 2017-06-28 16:31:32,593 DEBUG [ProcExecWrkr-1] backup.HFileArchiver(114): ARCHIVING hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/.tmp/data/default/testReplicaGetWithPrimaryAndMetaDown/da78e90b99801a7d916ba59be21ad54b 2017-06-28 16:31:32,594 DEBUG [ProcExecWrkr-1] backup.HFileArchiver(149): Region directory hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/.tmp/data/default/testReplicaGetWithPrimaryAndMetaDown/da78e90b99801a7d916ba59be21ad54b empty. 2017-06-28 16:31:32,595 DEBUG [ProcExecWrkr-1] backup.HFileArchiver(469): Failed to delete region directory:hdfs://localhost:59739/user/jenkins/test-data/3e13df75-f4b3-46f1-9bc5-eb7cad2a2001/.tmp/data/default/testReplicaGetWithPrimaryAndMetaDown/da78e90b99801a7d916ba59be21ad54b 2017-06-28 16:31:32,595 DEBUG [ProcExecWrkr-1] procedure.DeleteTableProcedure(317): Table 'testReplicaGetWithPrimaryAndMetaDown' archived! 2017-06-28 16:31:32,596 INFO [IPC Server handler 8 on 59739] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741858_1034 127.0.0.1:43805 127.0.0.1:36908 127.0.0.1:40636 2017-06-28 16:31:32,599 DEBUG [ProcExecWrkr-1] procedure.DeleteTableProcedure(121): delete 'testReplicaGetWithPrimaryAndMetaDown' descriptor 2017-06-28 16:31:32,599 DEBUG [ProcExecWrkr-1] procedure.DeleteTableProcedure(397): Removing 'testReplicaGetWithPrimaryAndMetaDown' descriptor. 2017-06-28 16:31:32,602 DEBUG [ProcExecWrkr-1] procedure.DeleteTableProcedure(126): delete 'testReplicaGetWithPrimaryAndMetaDown' assignment state 2017-06-28 16:31:32,602 DEBUG [ProcExecWrkr-1] procedure.DeleteTableProcedure(387): Removing 'testReplicaGetWithPrimaryAndMetaDown' from region states. 2017-06-28 16:31:32,603 DEBUG [ProcExecWrkr-1] hbase.MetaTableAccessor(1442): Delete{"totalColumns":1,"row":"testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1498667492602}]},"ts":9223372036854775807}, Delete{"totalColumns":1,"row":"testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1498667492602}]},"ts":9223372036854775807} 2017-06-28 16:31:32,607 INFO [ProcExecWrkr-1] hbase.MetaTableAccessor(1980): Deleted [{ENCODED => 1cf058a284bbd8c68aa7536052bfa697, NAME => 'testReplicaGetWithPrimaryAndMetaDown,,1498667486571.1cf058a284bbd8c68aa7536052bfa697.', STARTKEY => '', ENDKEY => ''}, {ENCODED => da78e90b99801a7d916ba59be21ad54b, NAME => 'testReplicaGetWithPrimaryAndMetaDown,,1498667486571_0001.da78e90b99801a7d916ba59be21ad54b.', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1}] 2017-06-28 16:31:32,607 DEBUG [ProcExecWrkr-1] procedure.DeleteTableProcedure(391): Marking 'testReplicaGetWithPrimaryAndMetaDown' as deleted. 2017-06-28 16:31:32,607 DEBUG [ProcExecWrkr-1] hbase.MetaTableAccessor(1442): Delete{"totalColumns":1,"row":"testReplicaGetWithPrimaryAndMetaDown","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":1498667492607}]},"ts":9223372036854775807} 2017-06-28 16:31:32,610 INFO [ProcExecWrkr-1] hbase.MetaTableAccessor(1793): Deleted table testReplicaGetWithPrimaryAndMetaDown state from META 2017-06-28 16:31:32,612 DEBUG [ProcExecWrkr-1] procedure.DeleteTableProcedure(132): delete 'testReplicaGetWithPrimaryAndMetaDown' completed 2017-06-28 16:31:32,614 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1167): Finished pid=68, state=SUCCESS; DeleteTableProcedure table=testReplicaGetWithPrimaryAndMetaDown in 170msec 2017-06-28 16:31:32,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40948] master.MasterRpcServices(1011): Checking to see if procedure is done procId=68 2017-06-28 16:31:32,650 INFO [main] client.HBaseAdmin$TableFuture(3584): Operation: DELETE, Table Name: default:testReplicaGetWithPrimaryAndMetaDown completed 2017-06-28 16:31:32,694 INFO [main] hbase.ResourceChecker(172): after: client.TestReplicaWithCluster#testReplicaGetWithPrimaryAndMetaDown Thread=596 (was 567) - Thread LEAK? -, OpenFileDescriptor=1381 (was 1379) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=250 (was 287), ProcessCount=269 (was 269), AvailableMemoryMB=17522 (was 17677) 2017-06-28 16:31:32,694 WARN [main] hbase.ResourceChecker(135): Thread=596 is superior to 500 2017-06-28 16:31:32,694 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1381 is superior to 1024