2014-08-24 06:15:05,117 INFO [main] hbase.ResourceChecker(147): before: master.TestZKLessAMOnCluster#testCloseHang Thread=548, OpenFileDescriptor=785, MaxFileDescriptor=65536, SystemLoadAverage=30, ProcessCount=152, AvailableMemoryMB=27870, ConnectionCount=10 2014-08-24 06:15:05,117 WARN [main] hbase.ResourceChecker(134): Thread=548 is superior to 500 2014-08-24 06:15:05,119 INFO [FifoRpcScheduler.handler7-thread-5] master.HMaster(1747): Client=apurtell//10.229.0.82 create 'testCloseHang', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2014-08-24 06:15:05,124 DEBUG [FifoRpcScheduler.handler7-thread-5] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testCloseHang/write-master:578260000000000 2014-08-24 06:15:05,130 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.CreateTableHandler(148): Create table testCloseHang 2014-08-24 06:15:05,138 INFO [IPC Server handler 2 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741864_1040{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:05,139 INFO [IPC Server handler 1 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741864_1040{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:05,139 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741864_1040 size 295 2014-08-24 06:15:05,140 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] util.FSTableDescriptors(640): Wrote descriptor into: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testCloseHang/.tabledesc/.tableinfo.0000000001 2014-08-24 06:15:05,142 INFO [RegionOpenAndInitThread-testCloseHang-1] regionserver.HRegion(4284): creating HRegion testCloseHang HTD == 'testCloseHang', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testCloseHang 2014-08-24 06:15:05,148 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741865_1041{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:05,149 INFO [IPC Server handler 5 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741865_1041{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:05,149 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741865_1041 size 46 2014-08-24 06:15:05,150 DEBUG [RegionOpenAndInitThread-testCloseHang-1] regionserver.HRegion(639): Instantiated testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. 2014-08-24 06:15:05,150 DEBUG [RegionOpenAndInitThread-testCloseHang-1] regionserver.HRegion(1109): Closing testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412.: disabling compactions & flushes 2014-08-24 06:15:05,150 DEBUG [RegionOpenAndInitThread-testCloseHang-1] regionserver.HRegion(1136): Updates disabled for region testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. 2014-08-24 06:15:05,150 INFO [RegionOpenAndInitThread-testCloseHang-1] regionserver.HRegion(1218): Closed testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. 2014-08-24 06:15:05,153 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] catalog.MetaEditor(278): Added 1 2014-08-24 06:15:05,153 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.AssignmentManager(2696): Bulk assigning 1 region(s) across 4 server(s), round-robin=true 2014-08-24 06:15:05,154 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.AssignmentManager(1597): Assigning 1 region(s) to ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,154 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStates(802): Transition {c82b5da19b9c763009afe987c395f412 state=OFFLINE, ts=1408860905153, server=null} to {c82b5da19b9c763009afe987c395f412 state=PENDING_OPEN, ts=1408860905154, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:05,154 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.GeneralBulkAssigner(177): Timeout-on-RIT=181000 2014-08-24 06:15:05,154 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStateStore(183): Updating row testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,157 INFO [PriorityRpcServer.handler=2,queue=0,port=51857] regionserver.HRegionServer(3718): Open testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. 2014-08-24 06:15:05,160 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.AssignmentManager(1777): Bulk assigning done for ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,160 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(4470): Opening region: {ENCODED => c82b5da19b9c763009afe987c395f412, NAME => 'testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412.', STARTKEY => '', ENDKEY => ''} 2014-08-24 06:15:05,160 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.GeneralBulkAssigner(153): bulk assigning total 1 regions to 4 servers, took 6ms, with 1 regions still in transition 2014-08-24 06:15:05,160 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.AssignmentManager(2703): Bulk assigning done 2014-08-24 06:15:05,160 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-1] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:05,160 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testCloseHang c82b5da19b9c763009afe987c395f412 2014-08-24 06:15:05,160 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(639): Instantiated testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. 2014-08-24 06:15:05,163 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testCloseHang/write-master:578260000000000 2014-08-24 06:15:05,163 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.CreateTableHandler(173): failed. null 2014-08-24 06:15:05,163 INFO [StoreOpener-c82b5da19b9c763009afe987c395f412-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:05,165 DEBUG [StoreOpener-c82b5da19b9c763009afe987c395f412-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testCloseHang/c82b5da19b9c763009afe987c395f412/FAMILY 2014-08-24 06:15:05,165 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testCloseHang/c82b5da19b9c763009afe987c395f412 2014-08-24 06:15:05,167 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(740): Onlined c82b5da19b9c763009afe987c395f412; next sequenceid=1 2014-08-24 06:15:05,167 INFO [PostOpenDeployTasks:c82b5da19b9c763009afe987c395f412] regionserver.HRegionServer(1796): Post open deploy tasks for region=testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. 2014-08-24 06:15:05,167 DEBUG [FifoRpcScheduler.handler7-thread-3] master.AssignmentManager(4068): Got transition OPENED for {c82b5da19b9c763009afe987c395f412 state=PENDING_OPEN, ts=1408860905154, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,167 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(802): Transition {c82b5da19b9c763009afe987c395f412 state=PENDING_OPEN, ts=1408860905154, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {c82b5da19b9c763009afe987c395f412 state=OPEN, ts=1408860905167, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:05,168 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStateStore(183): Updating row testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,170 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(359): Onlined c82b5da19b9c763009afe987c395f412 on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,172 INFO [PostOpenDeployTasks:c82b5da19b9c763009afe987c395f412] regionserver.HRegionServer(1827): Finished post open deploy task for testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. 2014-08-24 06:15:05,172 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] handler.OpenRegionHandler(185): Opened testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,239 INFO [Thread-644] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:05,239 DEBUG [Thread-644] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@589d7610 2014-08-24 06:15:05,241 DEBUG [Thread-644-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:05,241 DEBUG [Thread-644] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:05,242 DEBUG [Thread-644-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d0032 connected 2014-08-24 06:15:05,245 DEBUG [Thread-644] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@589d7610 2014-08-24 06:15:05,249 DEBUG [Thread-644] catalog.MetaEditor(238): Added testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:05,249 WARN [Thread-644] master.AssignmentManager(1918): Assigning a region not in region states: {ENCODED => 16764d6b00e27d16134c52e84f67e889, NAME => 'testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889.', STARTKEY => 'A', ENDKEY => 'Z'} 2014-08-24 06:15:05,250 DEBUG [Thread-644] master.AssignmentManager(2372): No previous transition plan found (or ignoring an existing plan) for testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889.; generated random plan=hri=testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889., src=, dest=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648; 4 (online=4, available=4) available servers, forceNewPlan=false 2014-08-24 06:15:05,250 INFO [Thread-644] master.AssignmentManager(2083): Assigning testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. to ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,250 INFO [Thread-644] master.RegionStates(802): Transition {16764d6b00e27d16134c52e84f67e889 state=OFFLINE, ts=1408860905249, server=null} to {16764d6b00e27d16134c52e84f67e889 state=PENDING_OPEN, ts=1408860905250, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:05,250 INFO [Thread-644] master.RegionStateStore(183): Updating row testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,252 INFO [PriorityRpcServer.handler=2,queue=0,port=51857] regionserver.HRegionServer(3718): Open testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:05,254 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(4470): Opening region: {ENCODED => 16764d6b00e27d16134c52e84f67e889, NAME => 'testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889.', STARTKEY => 'A', ENDKEY => 'Z'} 2014-08-24 06:15:05,254 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-2] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:05,254 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testCloseHang 16764d6b00e27d16134c52e84f67e889 2014-08-24 06:15:05,254 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(639): Instantiated testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:05,255 WARN [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegionFileSystem(788): .regioninfo file not found for region: 16764d6b00e27d16134c52e84f67e889 2014-08-24 06:15:05,261 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741866_1042{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:05,262 INFO [IPC Server handler 7 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741866_1042{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:05,262 INFO [IPC Server handler 2 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741866_1042 size 48 2014-08-24 06:15:05,267 INFO [StoreOpener-16764d6b00e27d16134c52e84f67e889-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:05,268 DEBUG [StoreOpener-16764d6b00e27d16134c52e84f67e889-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testCloseHang/16764d6b00e27d16134c52e84f67e889/FAMILY 2014-08-24 06:15:05,268 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testCloseHang/16764d6b00e27d16134c52e84f67e889 2014-08-24 06:15:05,269 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(740): Onlined 16764d6b00e27d16134c52e84f67e889; next sequenceid=1 2014-08-24 06:15:05,270 INFO [PostOpenDeployTasks:16764d6b00e27d16134c52e84f67e889] regionserver.HRegionServer(1796): Post open deploy tasks for region=testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:05,270 DEBUG [FifoRpcScheduler.handler7-thread-2] master.AssignmentManager(4068): Got transition OPENED for {16764d6b00e27d16134c52e84f67e889 state=PENDING_OPEN, ts=1408860905250, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,270 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(802): Transition {16764d6b00e27d16134c52e84f67e889 state=PENDING_OPEN, ts=1408860905250, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {16764d6b00e27d16134c52e84f67e889 state=OPEN, ts=1408860905270, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:05,270 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStateStore(183): Updating row testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,273 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(359): Onlined 16764d6b00e27d16134c52e84f67e889 on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,273 DEBUG [Thread-644] master.AssignmentManager(2420): Starting unassign of testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. (offlining), current state: {16764d6b00e27d16134c52e84f67e889 state=OPEN, ts=1408860905270, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:05,273 INFO [Thread-644] master.RegionStates(802): Transition {16764d6b00e27d16134c52e84f67e889 state=OPEN, ts=1408860905270, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {16764d6b00e27d16134c52e84f67e889 state=PENDING_CLOSE, ts=1408860905273, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:05,273 INFO [Thread-644] master.RegionStateStore(183): Updating row testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. with state=PENDING_CLOSE 2014-08-24 06:15:05,275 INFO [PriorityRpcServer.handler=3,queue=0,port=51857] regionserver.HRegionServer(3859): Close 16764d6b00e27d16134c52e84f67e889, via zk=no, znode version=-1, on null 2014-08-24 06:15:05,276 INFO [PriorityRpcServer.handler=3,queue=0,port=51857] regionserver.HRegionServer(2637): Received CLOSE for the region:16764d6b00e27d16134c52e84f67e889 , which we are already trying to OPEN. Cancelling OPENING. 2014-08-24 06:15:05,276 INFO [PriorityRpcServer.handler=3,queue=0,port=51857] regionserver.HRegionServer(2649): The opening previously in progress has been cancelled by a CLOSE request. 2014-08-24 06:15:05,276 INFO [PostOpenDeployTasks:16764d6b00e27d16134c52e84f67e889] regionserver.HRegionServer(1827): Finished post open deploy task for testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:05,276 DEBUG [Thread-644] master.AssignmentManager(1847): Offline testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889., it's not any more on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: The region 16764d6b00e27d16134c52e84f67e889 was opening but not yet served. Opening is cancelled. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2651) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3862) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testCloseHang(TestAssignmentManagerOnCluster.java:616) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException): org.apache.hadoop.hbase.NotServingRegionException: The region 16764d6b00e27d16134c52e84f67e889 was opening but not yet served. Opening is cancelled. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2651) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3862) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:05,276 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(1109): Closing testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889.: disabling compactions & flushes 2014-08-24 06:15:05,276 INFO [Thread-644] master.RegionStates(802): Transition {16764d6b00e27d16134c52e84f67e889 state=PENDING_CLOSE, ts=1408860905273, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {16764d6b00e27d16134c52e84f67e889 state=OFFLINE, ts=1408860905276, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:05,276 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(1136): Updates disabled for region testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:05,276 INFO [Thread-644] master.RegionStateStore(183): Updating row testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. with state=OFFLINE 2014-08-24 06:15:05,277 INFO [StoreCloserThread-testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:05,279 INFO [Thread-644] master.RegionStates(455): Offlined 16764d6b00e27d16134c52e84f67e889 from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,279 DEBUG [Thread-644] master.AssignmentManager(2372): No previous transition plan found (or ignoring an existing plan) for testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889.; generated random plan=hri=testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889., src=, dest=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571; 4 (online=4, available=4) available servers, forceNewPlan=false 2014-08-24 06:15:05,279 INFO [Thread-644] master.AssignmentManager(2083): Assigning testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,279 INFO [Thread-644] master.RegionStates(802): Transition {16764d6b00e27d16134c52e84f67e889 state=OFFLINE, ts=1408860905276, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {16764d6b00e27d16134c52e84f67e889 state=PENDING_OPEN, ts=1408860905279, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:05,280 INFO [Thread-644] master.RegionStateStore(183): Updating row testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,282 INFO [PriorityRpcServer.handler=7,queue=0,port=58042] regionserver.HRegionServer(3718): Open testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:05,285 DEBUG [Thread-644] master.AssignmentManager(2420): Starting unassign of testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. (offlining), current state: {16764d6b00e27d16134c52e84f67e889 state=PENDING_OPEN, ts=1408860905279, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:05,285 DEBUG [Thread-644] master.AssignmentManager(2507): Attempting to unassign testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. but it is already in transition (PENDING_OPEN, force=true) 2014-08-24 06:15:05,285 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(4470): Opening region: {ENCODED => 16764d6b00e27d16134c52e84f67e889, NAME => 'testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889.', STARTKEY => 'A', ENDKEY => 'Z'} 2014-08-24 06:15:05,286 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-1] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:05,286 INFO [Thread-644] client.HBaseAdmin$5(917): Started disable of testCloseHang 2014-08-24 06:15:05,286 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testCloseHang 16764d6b00e27d16134c52e84f67e889 2014-08-24 06:15:05,286 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(639): Instantiated testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:05,286 INFO [FifoRpcScheduler.handler7-thread-4] master.HMaster(1979): Client=apurtell//10.229.0.82 disable testCloseHang 2014-08-24 06:15:05,288 DEBUG [FifoRpcScheduler.handler7-thread-4] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testCloseHang/write-master:578260000000001 2014-08-24 06:15:05,289 INFO [StoreOpener-16764d6b00e27d16134c52e84f67e889-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:05,290 DEBUG [StoreOpener-16764d6b00e27d16134c52e84f67e889-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testCloseHang/16764d6b00e27d16134c52e84f67e889/FAMILY 2014-08-24 06:15:05,291 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testCloseHang/16764d6b00e27d16134c52e84f67e889 2014-08-24 06:15:05,292 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(740): Onlined 16764d6b00e27d16134c52e84f67e889; next sequenceid=1 2014-08-24 06:15:05,293 INFO [PostOpenDeployTasks:16764d6b00e27d16134c52e84f67e889] regionserver.HRegionServer(1796): Post open deploy tasks for region=testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:05,293 DEBUG [FifoRpcScheduler.handler7-thread-5] master.AssignmentManager(4068): Got transition OPENED for {16764d6b00e27d16134c52e84f67e889 state=PENDING_OPEN, ts=1408860905279, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,293 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(802): Transition {16764d6b00e27d16134c52e84f67e889 state=PENDING_OPEN, ts=1408860905279, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {16764d6b00e27d16134c52e84f67e889 state=OPEN, ts=1408860905293, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:05,293 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStateStore(183): Updating row testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,294 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(130): Attempting to disable table testCloseHang 2014-08-24 06:15:05,295 INFO [Thread-644] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:05,296 DEBUG [Thread-644] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@17bf6625 2014-08-24 06:15:05,296 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(174): Offlining 1 regions. 2014-08-24 06:15:05,296 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2420): Starting unassign of testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. (offlining), current state: {c82b5da19b9c763009afe987c395f412 state=OPEN, ts=1408860905167, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:05,296 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(802): Transition {c82b5da19b9c763009afe987c395f412 state=OPEN, ts=1408860905167, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {c82b5da19b9c763009afe987c395f412 state=PENDING_CLOSE, ts=1408860905296, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:05,297 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStateStore(183): Updating row testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. with state=PENDING_CLOSE 2014-08-24 06:15:05,297 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(359): Onlined 16764d6b00e27d16134c52e84f67e889 on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,297 DEBUG [Thread-644-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:05,297 DEBUG [Thread-644] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:05,298 DEBUG [Thread-644-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d0033 connected 2014-08-24 06:15:05,299 INFO [PriorityRpcServer.handler=4,queue=0,port=51857] regionserver.HRegionServer(3859): Close c82b5da19b9c763009afe987c395f412, via zk=no, znode version=-1, on null 2014-08-24 06:15:05,299 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] handler.CloseRegionHandler(128): Processing close of testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. 2014-08-24 06:15:05,299 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 for region testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. 2014-08-24 06:15:05,301 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(1109): Closing testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412.: disabling compactions & flushes 2014-08-24 06:15:05,301 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(1136): Updates disabled for region testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. 2014-08-24 06:15:05,301 INFO [StoreCloserThread-testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:05,301 DEBUG [AM.-pool37-t21] master.AssignmentManager(2420): Starting unassign of testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. (offlining), current state: {16764d6b00e27d16134c52e84f67e889 state=OPEN, ts=1408860905293, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:05,301 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:05,301 INFO [PostOpenDeployTasks:16764d6b00e27d16134c52e84f67e889] regionserver.HRegionServer(1827): Finished post open deploy task for testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:05,301 INFO [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(1218): Closed testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. 2014-08-24 06:15:05,301 INFO [AM.-pool37-t21] master.RegionStates(802): Transition {16764d6b00e27d16134c52e84f67e889 state=OPEN, ts=1408860905293, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {16764d6b00e27d16134c52e84f67e889 state=PENDING_CLOSE, ts=1408860905301, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:05,302 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] handler.OpenRegionHandler(185): Opened testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,302 INFO [AM.-pool37-t21] master.RegionStateStore(183): Updating row testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. with state=PENDING_CLOSE 2014-08-24 06:15:05,302 DEBUG [FifoRpcScheduler.handler7-thread-3] master.AssignmentManager(4068): Got transition CLOSED for {c82b5da19b9c763009afe987c395f412 state=PENDING_CLOSE, ts=1408860905296, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,304 DEBUG [Thread-644] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@17bf6625 2014-08-24 06:15:05,304 INFO [PriorityRpcServer.handler=8,queue=0,port=58042] regionserver.HRegionServer(3859): Close 16764d6b00e27d16134c52e84f67e889, via zk=no, znode version=-1, on null 2014-08-24 06:15:05,304 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] handler.CloseRegionHandler(128): Processing close of testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:05,305 DEBUG [AM.-pool37-t21] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 for region testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:05,305 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(1109): Closing testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889.: disabling compactions & flushes 2014-08-24 06:15:05,305 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(1136): Updates disabled for region testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:05,305 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(802): Transition {c82b5da19b9c763009afe987c395f412 state=PENDING_CLOSE, ts=1408860905296, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {c82b5da19b9c763009afe987c395f412 state=OFFLINE, ts=1408860905305, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:05,305 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStateStore(183): Updating row testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. with state=OFFLINE 2014-08-24 06:15:05,305 INFO [StoreCloserThread-testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:05,305 DEBUG [Thread-644] client.HBaseAdmin(956): Sleeping= 100ms, waiting for all regions to be disabled in testCloseHang 2014-08-24 06:15:05,306 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:05,306 INFO [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(1218): Closed testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:05,306 DEBUG [FifoRpcScheduler.handler7-thread-2] master.AssignmentManager(4068): Got transition CLOSED for {16764d6b00e27d16134c52e84f67e889 state=PENDING_CLOSE, ts=1408860905301, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,307 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(455): Offlined c82b5da19b9c763009afe987c395f412 from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,308 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] handler.CloseRegionHandler(182): Closed testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. 2014-08-24 06:15:05,308 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(802): Transition {16764d6b00e27d16134c52e84f67e889 state=PENDING_CLOSE, ts=1408860905301, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {16764d6b00e27d16134c52e84f67e889 state=OFFLINE, ts=1408860905308, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:05,308 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStateStore(183): Updating row testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. with state=OFFLINE 2014-08-24 06:15:05,310 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(455): Offlined 16764d6b00e27d16134c52e84f67e889 from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,311 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] handler.CloseRegionHandler(182): Closed testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:05,346 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler$BulkDisabler(230): Disable waiting until done; 300000 ms remaining; [] 2014-08-24 06:15:05,348 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(190): Disabled table, testCloseHang, is done=true 2014-08-24 06:15:05,349 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testCloseHang/write-master:578260000000001 2014-08-24 06:15:05,406 INFO [Thread-644] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:05,407 DEBUG [Thread-644] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@76ec071a 2014-08-24 06:15:05,408 DEBUG [Thread-644-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:05,408 DEBUG [Thread-644] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:05,409 DEBUG [Thread-644-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d0034 connected 2014-08-24 06:15:05,413 DEBUG [Thread-644] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@76ec071a 2014-08-24 06:15:05,414 INFO [Thread-644] client.HBaseAdmin(971): Disabled testCloseHang 2014-08-24 06:15:05,415 INFO [FifoRpcScheduler.handler7-thread-1] master.HMaster(1816): Client=apurtell//10.229.0.82 delete testCloseHang 2014-08-24 06:15:05,417 DEBUG [FifoRpcScheduler.handler7-thread-1] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testCloseHang/write-master:578260000000002 2014-08-24 06:15:05,424 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.TableEventHandler(123): Handling table operation C_M_DELETE_TABLE on table testCloseHang 2014-08-24 06:15:05,429 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(93): Deleting regions from META 2014-08-24 06:15:05,431 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] catalog.MetaEditor(495): Deleted [{ENCODED => c82b5da19b9c763009afe987c395f412, NAME => 'testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 16764d6b00e27d16134c52e84f67e889, NAME => 'testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889.', STARTKEY => 'A', ENDKEY => 'Z'}] 2014-08-24 06:15:05,433 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testCloseHang,,1408860905119.c82b5da19b9c763009afe987c395f412. from FS 2014-08-24 06:15:05,433 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testCloseHang/c82b5da19b9c763009afe987c395f412 2014-08-24 06:15:05,435 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testCloseHang/c82b5da19b9c763009afe987c395f412/FAMILY] 2014-08-24 06:15:05,438 INFO [IPC Server handler 1 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741865_1041 127.0.0.1:42916 127.0.0.1:60268 127.0.0.1:45654 2014-08-24 06:15:05,438 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testCloseHang/c82b5da19b9c763009afe987c395f412 2014-08-24 06:15:05,438 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. from FS 2014-08-24 06:15:05,439 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testCloseHang/16764d6b00e27d16134c52e84f67e889 2014-08-24 06:15:05,440 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testCloseHang/16764d6b00e27d16134c52e84f67e889/FAMILY] 2014-08-24 06:15:05,443 INFO [IPC Server handler 2 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741866_1042 127.0.0.1:45654 127.0.0.1:42916 127.0.0.1:60268 2014-08-24 06:15:05,444 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testCloseHang/16764d6b00e27d16134c52e84f67e889 2014-08-24 06:15:05,444 INFO [IPC Server handler 1 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741864_1040 127.0.0.1:60268 127.0.0.1:42916 127.0.0.1:45654 2014-08-24 06:15:05,444 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(114): Table 'testCloseHang' archived! 2014-08-24 06:15:05,444 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(117): Removing 'testCloseHang' descriptor. 2014-08-24 06:15:05,445 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(121): Removing 'testCloseHang' from region states. 2014-08-24 06:15:05,445 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(125): Marking 'testCloseHang' as deleted. 2014-08-24 06:15:05,447 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testCloseHang/write-master:578260000000002 2014-08-24 06:15:05,528 INFO [Thread-644] client.HBaseAdmin(705): Deleted testCloseHang 2014-08-24 06:15:05,545 INFO [main] hbase.ResourceChecker(171): after: master.TestZKLessAMOnCluster#testCloseHang Thread=549 (was 548) - Thread LEAK? -, OpenFileDescriptor=781 (was 785), MaxFileDescriptor=65536 (was 65536), SystemLoadAverage=30 (was 30), ProcessCount=152 (was 152), AvailableMemoryMB=27870 (was 27870), ConnectionCount=10 (was 10) 2014-08-24 06:15:05,545 WARN [main] hbase.ResourceChecker(134): Thread=549 is superior to 500 2014-08-24 06:15:05,558 INFO [main] hbase.ResourceChecker(147): before: master.TestZKLessAMOnCluster#testAssignRacingWithSSH Thread=549, OpenFileDescriptor=781, MaxFileDescriptor=65536, SystemLoadAverage=30, ProcessCount=152, AvailableMemoryMB=27869, ConnectionCount=10 2014-08-24 06:15:05,559 WARN [main] hbase.ResourceChecker(134): Thread=549 is superior to 500 2014-08-24 06:15:05,560 INFO [FifoRpcScheduler.handler7-thread-2] master.HMaster(1747): Client=apurtell//10.229.0.82 create 'testAssignRacingWithSSH', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2014-08-24 06:15:05,565 DEBUG [FifoRpcScheduler.handler7-thread-2] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testAssignRacingWithSSH/write-master:578260000000000 2014-08-24 06:15:05,573 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.CreateTableHandler(148): Create table testAssignRacingWithSSH 2014-08-24 06:15:05,581 INFO [IPC Server handler 1 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741867_1043{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:05,584 INFO [IPC Server handler 3 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741867_1043{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:05,584 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741867_1043 size 305 2014-08-24 06:15:05,586 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] util.FSTableDescriptors(640): Wrote descriptor into: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testAssignRacingWithSSH/.tabledesc/.tableinfo.0000000001 2014-08-24 06:15:05,587 INFO [RegionOpenAndInitThread-testAssignRacingWithSSH-1] regionserver.HRegion(4284): creating HRegion testAssignRacingWithSSH HTD == 'testAssignRacingWithSSH', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testAssignRacingWithSSH 2014-08-24 06:15:05,594 INFO [IPC Server handler 5 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741868_1044{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:05,594 INFO [IPC Server handler 8 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741868_1044{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:05,595 INFO [IPC Server handler 7 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741868_1044 size 56 2014-08-24 06:15:05,595 DEBUG [RegionOpenAndInitThread-testAssignRacingWithSSH-1] regionserver.HRegion(639): Instantiated testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. 2014-08-24 06:15:05,595 DEBUG [RegionOpenAndInitThread-testAssignRacingWithSSH-1] regionserver.HRegion(1109): Closing testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840.: disabling compactions & flushes 2014-08-24 06:15:05,595 DEBUG [RegionOpenAndInitThread-testAssignRacingWithSSH-1] regionserver.HRegion(1136): Updates disabled for region testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. 2014-08-24 06:15:05,595 INFO [RegionOpenAndInitThread-testAssignRacingWithSSH-1] regionserver.HRegion(1218): Closed testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. 2014-08-24 06:15:05,599 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] catalog.MetaEditor(278): Added 1 2014-08-24 06:15:05,599 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.AssignmentManager(2696): Bulk assigning 1 region(s) across 4 server(s), round-robin=true 2014-08-24 06:15:05,600 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.GeneralBulkAssigner(177): Timeout-on-RIT=181000 2014-08-24 06:15:05,600 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-3] master.AssignmentManager(1597): Assigning 1 region(s) to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,600 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-3] master.RegionStates(802): Transition {194f89dd1ff2fa120355638b0291f840 state=OFFLINE, ts=1408860905599, server=null} to {194f89dd1ff2fa120355638b0291f840 state=PENDING_OPEN, ts=1408860905600, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:05,600 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-3] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,602 INFO [PriorityRpcServer.handler=9,queue=0,port=58042] regionserver.HRegionServer(3718): Open testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. 2014-08-24 06:15:05,606 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-3] master.AssignmentManager(1777): Bulk assigning done for ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,606 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(4470): Opening region: {ENCODED => 194f89dd1ff2fa120355638b0291f840, NAME => 'testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840.', STARTKEY => '', ENDKEY => ''} 2014-08-24 06:15:05,606 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.GeneralBulkAssigner(153): bulk assigning total 1 regions to 4 servers, took 6ms, with 1 regions still in transition 2014-08-24 06:15:05,606 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.AssignmentManager(2703): Bulk assigning done 2014-08-24 06:15:05,606 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-2] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:05,606 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testAssignRacingWithSSH 194f89dd1ff2fa120355638b0291f840 2014-08-24 06:15:05,606 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(639): Instantiated testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. 2014-08-24 06:15:05,609 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testAssignRacingWithSSH/write-master:578260000000000 2014-08-24 06:15:05,609 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.CreateTableHandler(173): failed. null 2014-08-24 06:15:05,609 INFO [StoreOpener-194f89dd1ff2fa120355638b0291f840-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:05,610 DEBUG [StoreOpener-194f89dd1ff2fa120355638b0291f840-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testAssignRacingWithSSH/194f89dd1ff2fa120355638b0291f840/FAMILY 2014-08-24 06:15:05,611 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testAssignRacingWithSSH/194f89dd1ff2fa120355638b0291f840 2014-08-24 06:15:05,612 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(740): Onlined 194f89dd1ff2fa120355638b0291f840; next sequenceid=1 2014-08-24 06:15:05,612 INFO [PostOpenDeployTasks:194f89dd1ff2fa120355638b0291f840] regionserver.HRegionServer(1796): Post open deploy tasks for region=testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. 2014-08-24 06:15:05,613 DEBUG [FifoRpcScheduler.handler7-thread-1] master.AssignmentManager(4068): Got transition OPENED for {194f89dd1ff2fa120355638b0291f840 state=PENDING_OPEN, ts=1408860905600, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,613 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(802): Transition {194f89dd1ff2fa120355638b0291f840 state=PENDING_OPEN, ts=1408860905600, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {194f89dd1ff2fa120355638b0291f840 state=OPEN, ts=1408860905613, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:05,613 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,615 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(359): Onlined 194f89dd1ff2fa120355638b0291f840 on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,617 INFO [PostOpenDeployTasks:194f89dd1ff2fa120355638b0291f840] regionserver.HRegionServer(1827): Finished post open deploy task for testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. 2014-08-24 06:15:05,617 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-2] handler.OpenRegionHandler(185): Opened testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,686 INFO [Thread-670] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:05,686 DEBUG [Thread-670] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@613a01ca 2014-08-24 06:15:05,688 DEBUG [Thread-670-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:05,688 DEBUG [Thread-670] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:05,689 DEBUG [Thread-670-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d0035 connected 2014-08-24 06:15:05,694 DEBUG [Thread-670] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@613a01ca 2014-08-24 06:15:05,697 DEBUG [Thread-670] catalog.MetaEditor(238): Added testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:05,697 WARN [Thread-670] master.AssignmentManager(1918): Assigning a region not in region states: {ENCODED => e4055a2384e5839468e67fd9cd4bd3a0, NAME => 'testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0.', STARTKEY => 'A', ENDKEY => 'Z'} 2014-08-24 06:15:05,697 DEBUG [Thread-670] master.AssignmentManager(2372): No previous transition plan found (or ignoring an existing plan) for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0.; generated random plan=hri=testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., src=, dest=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648; 4 (online=4, available=4) available servers, forceNewPlan=false 2014-08-24 06:15:05,698 INFO [Thread-670] master.AssignmentManager(2083): Assigning testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. to ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,698 INFO [Thread-670] master.RegionStates(802): Transition {e4055a2384e5839468e67fd9cd4bd3a0 state=OFFLINE, ts=1408860905697, server=null} to {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_OPEN, ts=1408860905698, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:05,698 INFO [Thread-670] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,700 INFO [PriorityRpcServer.handler=8,queue=0,port=51857] regionserver.HRegionServer(3718): Open testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:05,703 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(4470): Opening region: {ENCODED => e4055a2384e5839468e67fd9cd4bd3a0, NAME => 'testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0.', STARTKEY => 'A', ENDKEY => 'Z'} 2014-08-24 06:15:05,703 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-0] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:05,704 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testAssignRacingWithSSH e4055a2384e5839468e67fd9cd4bd3a0 2014-08-24 06:15:05,704 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(639): Instantiated testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:05,704 WARN [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.HRegionFileSystem(788): .regioninfo file not found for region: e4055a2384e5839468e67fd9cd4bd3a0 2014-08-24 06:15:05,711 INFO [IPC Server handler 9 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741869_1045{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:05,711 INFO [IPC Server handler 2 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741869_1045{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:05,712 INFO [IPC Server handler 3 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741869_1045 size 58 2014-08-24 06:15:05,715 INFO [StoreOpener-e4055a2384e5839468e67fd9cd4bd3a0-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:05,716 DEBUG [StoreOpener-e4055a2384e5839468e67fd9cd4bd3a0-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testAssignRacingWithSSH/e4055a2384e5839468e67fd9cd4bd3a0/FAMILY 2014-08-24 06:15:05,717 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testAssignRacingWithSSH/e4055a2384e5839468e67fd9cd4bd3a0 2014-08-24 06:15:05,718 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(740): Onlined e4055a2384e5839468e67fd9cd4bd3a0; next sequenceid=1 2014-08-24 06:15:05,718 INFO [PostOpenDeployTasks:e4055a2384e5839468e67fd9cd4bd3a0] regionserver.HRegionServer(1796): Post open deploy tasks for region=testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:05,719 DEBUG [FifoRpcScheduler.handler7-thread-5] master.AssignmentManager(4068): Got transition OPENED for {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_OPEN, ts=1408860905698, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,719 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(802): Transition {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_OPEN, ts=1408860905698, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {e4055a2384e5839468e67fd9cd4bd3a0 state=OPEN, ts=1408860905719, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:05,719 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,721 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(359): Onlined e4055a2384e5839468e67fd9cd4bd3a0 on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,721 INFO [Thread-670] master.HMaster(1717): Client=null/null move hri=testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., src=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648, dest=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519, running balancer 2014-08-24 06:15:05,721 DEBUG [Thread-670] master.AssignmentManager(2420): Starting unassign of testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. (offlining), current state: {e4055a2384e5839468e67fd9cd4bd3a0 state=OPEN, ts=1408860905719, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:05,721 INFO [Thread-670] master.RegionStates(802): Transition {e4055a2384e5839468e67fd9cd4bd3a0 state=OPEN, ts=1408860905719, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_CLOSE, ts=1408860905721, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:05,721 INFO [Thread-670] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. with state=PENDING_CLOSE 2014-08-24 06:15:05,723 INFO [PostOpenDeployTasks:e4055a2384e5839468e67fd9cd4bd3a0] regionserver.HRegionServer(1827): Finished post open deploy task for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:05,723 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-0] handler.OpenRegionHandler(185): Opened testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,724 INFO [PriorityRpcServer.handler=2,queue=0,port=51857] regionserver.HRegionServer(3859): Close e4055a2384e5839468e67fd9cd4bd3a0, via zk=no, znode version=-1, on ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,724 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] handler.CloseRegionHandler(128): Processing close of testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:05,724 DEBUG [Thread-670] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 for region testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:05,725 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(1109): Closing testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0.: disabling compactions & flushes 2014-08-24 06:15:05,725 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(1136): Updates disabled for region testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:05,725 INFO [StoreCloserThread-testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:05,726 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:05,726 INFO [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(1218): Closed testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:05,726 INFO [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] regionserver.HRegionServer(4527): Adding moved region record: e4055a2384e5839468e67fd9cd4bd3a0 to ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519:57246 as of 1 2014-08-24 06:15:05,727 DEBUG [FifoRpcScheduler.handler7-thread-4] master.AssignmentManager(4068): Got transition CLOSED for {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_CLOSE, ts=1408860905721, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,729 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(802): Transition {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_CLOSE, ts=1408860905721, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {e4055a2384e5839468e67fd9cd4bd3a0 state=CLOSED, ts=1408860905729, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:05,729 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. with state=CLOSED 2014-08-24 06:15:05,731 DEBUG [AM.-pool37-t22] master.AssignmentManager(2342): Found an existing plan for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. destination server is ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 accepted as a dest server = true 2014-08-24 06:15:05,731 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] handler.CloseRegionHandler(182): Closed testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:05,731 DEBUG [AM.-pool37-t22] master.AssignmentManager(2381): Using pre-existing plan for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0.; plan=hri=testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., src=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648, dest=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,731 INFO [AM.-pool37-t22] master.AssignmentManager(2083): Assigning testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. to ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,731 INFO [AM.-pool37-t22] master.RegionStates(802): Transition {e4055a2384e5839468e67fd9cd4bd3a0 state=CLOSED, ts=1408860905729, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_OPEN, ts=1408860905731, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} 2014-08-24 06:15:05,732 INFO [AM.-pool37-t22] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,734 INFO [PriorityRpcServer.handler=4,queue=0,port=57246] regionserver.HRegionServer(3718): Open testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:05,737 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:57246-2] regionserver.HRegion(4470): Opening region: {ENCODED => e4055a2384e5839468e67fd9cd4bd3a0, NAME => 'testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0.', STARTKEY => 'A', ENDKEY => 'Z'} 2014-08-24 06:15:05,737 INFO [RS_OPEN_REGION-ip-10-229-0-82:57246-2] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:05,737 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:57246-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testAssignRacingWithSSH e4055a2384e5839468e67fd9cd4bd3a0 2014-08-24 06:15:05,738 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:57246-2] regionserver.HRegion(639): Instantiated testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:05,740 INFO [StoreOpener-e4055a2384e5839468e67fd9cd4bd3a0-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:05,741 DEBUG [StoreOpener-e4055a2384e5839468e67fd9cd4bd3a0-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testAssignRacingWithSSH/e4055a2384e5839468e67fd9cd4bd3a0/FAMILY 2014-08-24 06:15:05,742 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:57246-2] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testAssignRacingWithSSH/e4055a2384e5839468e67fd9cd4bd3a0 2014-08-24 06:15:05,743 INFO [RS_OPEN_REGION-ip-10-229-0-82:57246-2] regionserver.HRegion(740): Onlined e4055a2384e5839468e67fd9cd4bd3a0; next sequenceid=1 2014-08-24 06:15:05,743 INFO [PostOpenDeployTasks:e4055a2384e5839468e67fd9cd4bd3a0] regionserver.HRegionServer(1796): Post open deploy tasks for region=testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:05,743 DEBUG [FifoRpcScheduler.handler7-thread-3] master.AssignmentManager(4068): Got transition OPENED for {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_OPEN, ts=1408860905731, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} from ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,743 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(802): Transition {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_OPEN, ts=1408860905731, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} to {e4055a2384e5839468e67fd9cd4bd3a0 state=OPEN, ts=1408860905743, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} 2014-08-24 06:15:05,744 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,746 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(359): Onlined e4055a2384e5839468e67fd9cd4bd3a0 on ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,746 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(367): Offlined e4055a2384e5839468e67fd9cd4bd3a0 from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,746 INFO [Thread-670] hbase.MiniHBaseCluster(244): Killing ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,746 FATAL [Thread-670] regionserver.HRegionServer(1852): ABORTING region server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519: Simulated kill 2014-08-24 06:15:05,746 FATAL [Thread-670] regionserver.HRegionServer(1858): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver, org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2014-08-24 06:15:05,748 INFO [PostOpenDeployTasks:e4055a2384e5839468e67fd9cd4bd3a0] regionserver.HRegionServer(1827): Finished post open deploy task for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:05,748 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:57246-2] handler.OpenRegionHandler(185): Opened testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. on ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,751 ERROR [FifoRpcScheduler.handler7-thread-2] master.HMaster(1371): Region server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 reported a fatal error: ABORTING region server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519: Simulated kill 2014-08-24 06:15:05,752 INFO [Thread-670] regionserver.HRegionServer(1777): STOPPED: Simulated kill 2014-08-24 06:15:05,752 INFO [Thread-670] hbase.LocalHBaseCluster(261): Waiting on ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,753 INFO [RS:0;ip-10-229-0-82:57246] regionserver.SplitLogWorker(604): Sending interrupt to stop the worker thread 2014-08-24 06:15:05,753 INFO [RS:0;ip-10-229-0-82:57246] regionserver.HRegionServer(932): Stopping infoServer 2014-08-24 06:15:05,753 INFO [SplitLogWorker-ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519] regionserver.SplitLogWorker(295): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException 2014-08-24 06:15:05,753 INFO [SplitLogWorker-ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519] regionserver.SplitLogWorker(212): SplitLogWorker ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 exiting 2014-08-24 06:15:05,755 INFO [RS:0;ip-10-229-0-82:57246] log.Slf4jLog(67): Stopped SelectChannelConnector@0.0.0.0:0 2014-08-24 06:15:05,856 INFO [RS:0;ip-10-229-0-82:57246.logRoller] regionserver.LogRoller(120): LogRoller exiting. 2014-08-24 06:15:05,856 INFO [RS:0;ip-10-229-0-82:57246.nonceCleaner] hbase.Chore(93): RS:0;ip-10-229-0-82:57246.nonceCleaner exiting 2014-08-24 06:15:05,856 INFO [RS:0;ip-10-229-0-82:57246.compactionChecker] hbase.Chore(93): RS:0;ip-10-229-0-82:57246.compactionChecker exiting 2014-08-24 06:15:05,856 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(274): MemStoreFlusher.1 exiting 2014-08-24 06:15:05,856 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(274): MemStoreFlusher.0 exiting 2014-08-24 06:15:05,856 INFO [RS:0;ip-10-229-0-82:57246] snapshot.RegionServerSnapshotManager(133): Stopping RegionServerSnapshotManager abruptly. 2014-08-24 06:15:05,857 DEBUG [RS:0;ip-10-229-0-82:57246] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@7de37ee4 2014-08-24 06:15:05,857 INFO [RS:0;ip-10-229-0-82:57246] client.HConnectionManager$HConnectionImplementation(1850): Closing zookeeper sessionid=0x14806a73b1d0008 2014-08-24 06:15:05,858 DEBUG [RS:0;ip-10-229-0-82:57246-WAL.AsyncNotifier] wal.FSHLog$AsyncNotifier(1332): RS:0;ip-10-229-0-82:57246-WAL.AsyncNotifier interrupted while waiting for notification from AsyncSyncer thread 2014-08-24 06:15:05,858 INFO [RS:0;ip-10-229-0-82:57246-WAL.AsyncNotifier] wal.FSHLog$AsyncNotifier(1337): RS:0;ip-10-229-0-82:57246-WAL.AsyncNotifier exiting 2014-08-24 06:15:05,859 DEBUG [RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer0] wal.FSHLog$AsyncSyncer(1278): RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer0 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:05,859 INFO [RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer0] wal.FSHLog$AsyncSyncer(1283): RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer0 exiting 2014-08-24 06:15:05,859 DEBUG [RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer1] wal.FSHLog$AsyncSyncer(1278): RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer1 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:05,859 INFO [RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer1] wal.FSHLog$AsyncSyncer(1283): RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer1 exiting 2014-08-24 06:15:05,859 DEBUG [RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer2] wal.FSHLog$AsyncSyncer(1278): RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer2 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:05,859 INFO [RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer2] wal.FSHLog$AsyncSyncer(1283): RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer2 exiting 2014-08-24 06:15:05,859 DEBUG [RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer3] wal.FSHLog$AsyncSyncer(1278): RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer3 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:05,859 INFO [RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer3] wal.FSHLog$AsyncSyncer(1283): RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer3 exiting 2014-08-24 06:15:05,860 DEBUG [RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer4] wal.FSHLog$AsyncSyncer(1278): RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer4 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:05,860 INFO [RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer4] wal.FSHLog$AsyncSyncer(1283): RS:0;ip-10-229-0-82:57246-WAL.AsyncSyncer4 exiting 2014-08-24 06:15:05,860 DEBUG [RS:0;ip-10-229-0-82:57246-WAL.AsyncWriter] wal.FSHLog$AsyncWriter(1153): RS:0;ip-10-229-0-82:57246-WAL.AsyncWriter interrupted while waiting for newer writes added to local buffer 2014-08-24 06:15:05,860 INFO [RS:0;ip-10-229-0-82:57246-WAL.AsyncWriter] wal.FSHLog$AsyncWriter(1158): RS:0;ip-10-229-0-82:57246-WAL.AsyncWriter exiting 2014-08-24 06:15:05,860 DEBUG [RS:0;ip-10-229-0-82:57246] wal.FSHLog(938): Closing WAL writer in hdfs://localhost:59365/user/apurtell/hbase/WALs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,863 INFO [IPC Server handler 1 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741832_1008{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW]]} size 0 2014-08-24 06:15:05,863 INFO [IPC Server handler 3 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741832_1008{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW]]} size 0 2014-08-24 06:15:05,864 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741832_1008 size 303 2014-08-24 06:15:05,964 INFO [RS:0;ip-10-229-0-82:57246] regionserver.Leases(147): RS:0;ip-10-229-0-82:57246 closing leases 2014-08-24 06:15:05,964 INFO [RS:0;ip-10-229-0-82:57246] regionserver.Leases(150): RS:0;ip-10-229-0-82:57246 closed leases 2014-08-24 06:15:05,965 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:58042-0x14806a73b1d0002, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,965 DEBUG [Thread-434-EventThread] zookeeper.ZooKeeperWatcher(312): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,965 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,965 DEBUG [RS:0;ip-10-229-0-82:57246-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:57246-0x14806a73b1d0001, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,965 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,966 DEBUG [RS:0;ip-10-229-0-82:57246-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:57246-0x14806a73b1d0001, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2014-08-24 06:15:05,965 INFO [Thread-434-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519] 2014-08-24 06:15:05,967 INFO [Thread-434-EventThread] master.ServerManager(505): Master doesn't enable ServerShutdownHandler during initialization, delay expiring server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:05,967 DEBUG [Thread-434-EventThread] zookeeper.ZooKeeperWatcher(312): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2014-08-24 06:15:05,967 INFO [RS:0;ip-10-229-0-82:57246] regionserver.HRegionServer(1035): stopping server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519; zookeeper connection closed. 2014-08-24 06:15:05,967 INFO [RS:0;ip-10-229-0-82:57246] regionserver.HRegionServer(1038): RS:0;ip-10-229-0-82:57246 exiting 2014-08-24 06:15:05,967 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] zookeeper.ZKUtil(428): regionserver:58042-0x14806a73b1d0002, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,967 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5a296147] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(190): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5a296147 2014-08-24 06:15:05,967 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZKUtil(428): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,968 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZKUtil(428): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,968 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] zookeeper.ZKUtil(428): regionserver:58042-0x14806a73b1d0002, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,968 DEBUG [Thread-434-EventThread] zookeeper.ZKUtil(428): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,968 DEBUG [Thread-670] master.AssignmentManager(1924): Force region state offline {e4055a2384e5839468e67fd9cd4bd3a0 state=OPEN, ts=1408860905743, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} 2014-08-24 06:15:05,968 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZKUtil(428): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,968 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZKUtil(428): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,968 DEBUG [Thread-434-EventThread] zookeeper.ZKUtil(428): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,968 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.io.IOException: Unexpected closed connection for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=1 of 20 java.io.IOException: Unexpected closed connection at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1450) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:05,969 DEBUG [Thread-434-EventThread] zookeeper.ZKUtil(428): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:05,968 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] zookeeper.ZKUtil(428): regionserver:58042-0x14806a73b1d0002, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:05,969 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZKUtil(428): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:05,968 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZKUtil(428): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:05,969 INFO [RS:2;ip-10-229-0-82:55743-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(119): /hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 znode expired, triggering replicatorRemoved event 2014-08-24 06:15:05,969 INFO [RS:1;ip-10-229-0-82:58042-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(119): /hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 znode expired, triggering replicatorRemoved event 2014-08-24 06:15:05,970 DEBUG [Thread-434-EventThread] zookeeper.RegionServerTracker(91): RS node: /hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 data: PBUF�� 2014-08-24 06:15:05,970 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=2 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:05,970 INFO [RS:3;ip-10-229-0-82:51857-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(119): /hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 znode expired, triggering replicatorRemoved event 2014-08-24 06:15:05,971 DEBUG [Thread-434-EventThread] zookeeper.RegionServerTracker(91): RS node: /hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 data: PBUF�� 2014-08-24 06:15:05,971 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2014-08-24 06:15:05,971 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2014-08-24 06:15:05,971 DEBUG [Thread-434-EventThread] zookeeper.RegionServerTracker(91): RS node: /hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 data: PBUF�� 2014-08-24 06:15:05,971 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:58042-0x14806a73b1d0002, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2014-08-24 06:15:05,972 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZKUtil(428): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,972 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZKUtil(428): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,972 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] zookeeper.ZKUtil(428): regionserver:58042-0x14806a73b1d0002, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:05,972 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZKUtil(428): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,972 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZKUtil(428): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,973 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] zookeeper.ZKUtil(428): regionserver:58042-0x14806a73b1d0002, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:05,973 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZKUtil(428): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:05,973 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZKUtil(428): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:05,973 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] zookeeper.ZKUtil(428): regionserver:58042-0x14806a73b1d0002, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:06,218 INFO [RS:0;ip-10-229-0-82:57246.periodicFlusher] hbase.Chore(93): RS:0;ip-10-229-0-82:57246.periodicFlusher exiting 2014-08-24 06:15:06,219 INFO [RS:0;ip-10-229-0-82:57246.leaseChecker] regionserver.Leases(147): RS:0;ip-10-229-0-82:57246.leaseChecker closing leases 2014-08-24 06:15:06,219 INFO [RS:0;ip-10-229-0-82:57246.leaseChecker] regionserver.Leases(150): RS:0;ip-10-229-0-82:57246.leaseChecker closed leases 2014-08-24 06:15:06,277 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:06,277 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(1218): Closed testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889. 2014-08-24 06:15:06,278 DEBUG [FifoRpcScheduler.handler7-thread-1] master.AssignmentManager(4068): Got transition FAILED_OPEN for 16764d6b00e27d16134c52e84f67e889 from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:06,278 ERROR [FifoRpcScheduler.handler7-thread-1] master.AssignmentManager(4132): Failed to transtion region from null to FAILED_OPEN by ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648: 16764d6b00e27d16134c52e84f67e889 is not pending open on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:06,278 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegionServer(1992): Failed to transition {ENCODED => 16764d6b00e27d16134c52e84f67e889, NAME => 'testCloseHang,A,1408860905247.16764d6b00e27d16134c52e84f67e889.', STARTKEY => 'A', ENDKEY => 'Z'} to FAILED_OPEN: 16764d6b00e27d16134c52e84f67e889 is not pending open on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:07,972 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=3 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:07,973 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=4 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:08,812 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl(169): Moving ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519's hlogs to my queue 2014-08-24 06:15:08,817 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519/lock 2014-08-24 06:15:09,279 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl(169): Moving ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519's hlogs to my queue 2014-08-24 06:15:09,280 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl(239): Won't transfer the queue, another RS took care of it because of: KeeperErrorCode = NoNode for /hbase/replication/rs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519/lock 2014-08-24 06:15:09,674 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl(169): Moving ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519's hlogs to my queue 2014-08-24 06:15:09,675 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl(239): Won't transfer the queue, another RS took care of it because of: KeeperErrorCode = NoNode for /hbase/replication/rs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519/lock 2014-08-24 06:15:09,974 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=5 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:09,975 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=6 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:11,976 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=7 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:11,977 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=8 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:13,978 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=9 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:13,979 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=10 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:15,980 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=11 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:15,981 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=12 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:17,982 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=13 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:17,985 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=14 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:19,987 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=15 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:19,987 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=16 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:21,989 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=17 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:21,989 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=18 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:23,991 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=19 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:23,991 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=20 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:764) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:23,992 WARN [Thread-670] master.RegionStates(793): Failed to open/close e4055a2384e5839468e67fd9cd4bd3a0 on ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519, set to FAILED_CLOSE 2014-08-24 06:15:23,992 INFO [Thread-670] master.RegionStates(802): Transition {e4055a2384e5839468e67fd9cd4bd3a0 state=OPEN, ts=1408860905743, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} to {e4055a2384e5839468e67fd9cd4bd3a0 state=FAILED_CLOSE, ts=1408860923992, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} 2014-08-24 06:15:23,992 INFO [Thread-670] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. with state=FAILED_CLOSE 2014-08-24 06:15:23,995 INFO [Thread-670] master.AssignmentManager(1945): Skip assigning {ENCODED => e4055a2384e5839468e67fd9cd4bd3a0, NAME => 'testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0.', STARTKEY => 'A', ENDKEY => 'Z'}, we couldn't close it: {e4055a2384e5839468e67fd9cd4bd3a0 state=FAILED_CLOSE, ts=1408860923992, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} 2014-08-24 06:15:23,995 DEBUG [Thread-670] master.AssignmentManager(2420): Starting unassign of testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. (offlining), current state: {e4055a2384e5839468e67fd9cd4bd3a0 state=FAILED_CLOSE, ts=1408860923992, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} 2014-08-24 06:15:23,995 DEBUG [Thread-670] master.AssignmentManager(2499): Attempting to unassign testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. which is already FAILED_CLOSE but forcing to send a CLOSE RPC again 2014-08-24 06:15:23,995 INFO [Thread-670] master.RegionStates(802): Transition {e4055a2384e5839468e67fd9cd4bd3a0 state=FAILED_CLOSE, ts=1408860923992, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} to {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_CLOSE, ts=1408860923995, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} 2014-08-24 06:15:23,995 INFO [Thread-670] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. with state=PENDING_CLOSE 2014-08-24 06:15:25,999 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=1 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:26,000 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=2 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:28,001 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=3 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:28,002 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=4 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:30,004 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=5 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:30,004 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=6 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:32,006 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=7 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:32,006 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=8 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:34,008 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=9 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:34,008 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=10 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:36,010 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=11 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:36,010 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=12 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:38,012 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=13 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:38,012 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=14 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:40,014 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=15 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:40,014 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=16 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:42,016 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=17 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:42,016 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=18 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:44,018 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=19 of 20 org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82:57246 at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:44,018 INFO [Thread-670] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 returned java.net.ConnectException: Connection refused for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0., try=20 of 20 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578) at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868) at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testAssignRacingWithSSH(TestAssignmentManagerOnCluster.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2014-08-24 06:15:44,021 WARN [Thread-670] master.RegionStates(793): Failed to open/close e4055a2384e5839468e67fd9cd4bd3a0 on ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519, set to FAILED_CLOSE 2014-08-24 06:15:44,021 INFO [Thread-670] master.RegionStates(802): Transition {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_CLOSE, ts=1408860923998, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} to {e4055a2384e5839468e67fd9cd4bd3a0 state=FAILED_CLOSE, ts=1408860944021, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} 2014-08-24 06:15:44,021 INFO [Thread-670] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. with state=FAILED_CLOSE 2014-08-24 06:15:44,024 DEBUG [Thread-670] master.AssignmentManager(3378): based on AM, current region=hbase:meta,,1.1588230740 is on server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 server being checked: ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:44,025 DEBUG [Thread-670] master.ServerManager(547): Added=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 to dead servers, submitted shutdown handler to be executed meta=false 2014-08-24 06:15:44,025 INFO [Thread-670] master.AssignmentManager(3131): Waiting for e4055a2384e5839468e67fd9cd4bd3a0 to leave regions-in-transition, timeOut=-1 ms. 2014-08-24 06:15:44,032 INFO [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] handler.ServerShutdownHandler(194): Splitting logs for ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 before assignment. 2014-08-24 06:15:44,035 DEBUG [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.MasterFileSystem(328): Renamed region directory: hdfs://localhost:59365/user/apurtell/hbase/WALs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519-splitting 2014-08-24 06:15:44,035 INFO [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.SplitLogManager(1457): dead splitlog workers [ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519] 2014-08-24 06:15:44,037 DEBUG [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.SplitLogManager(342): Scheduling batch of logs to split 2014-08-24 06:15:44,038 INFO [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.SplitLogManager(344): started splitting 1 logs in [hdfs://localhost:59365/user/apurtell/hbase/WALs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519-splitting] 2014-08-24 06:15:44,043 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:58042-0x14806a73b1d0002, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2014-08-24 06:15:44,043 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2014-08-24 06:15:44,043 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] regionserver.SplitLogWorker(591): tasks arrived or departed 2014-08-24 06:15:44,043 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2014-08-24 06:15:44,043 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] regionserver.SplitLogWorker(591): tasks arrived or departed 2014-08-24 06:15:44,043 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] regionserver.SplitLogWorker(591): tasks arrived or departed 2014-08-24 06:15:44,044 DEBUG [Thread-434-EventThread] master.SplitLogManager(717): put up splitlog task at znode /hbase/splitWAL/WALs%2Fip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519-splitting%2Fip-10-229-0-82.us-west-2.compute.internal%252C57246%252C1408860896519.1408860897173 2014-08-24 06:15:44,045 INFO [SplitLogWorker-ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608] regionserver.SplitLogWorker(359): worker ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 acquired task /hbase/splitWAL/WALs%2Fip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519-splitting%2Fip-10-229-0-82.us-west-2.compute.internal%252C57246%252C1408860896519.1408860897173 2014-08-24 06:15:44,046 INFO [Thread-434-EventThread] master.SplitLogManager(832): task /hbase/splitWAL/WALs%2Fip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519-splitting%2Fip-10-229-0-82.us-west-2.compute.internal%252C57246%252C1408860896519.1408860897173 acquired by ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,068 INFO [RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0] wal.HLogSplitter(273): Splitting hlog: hdfs://localhost:59365/user/apurtell/hbase/WALs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519-splitting/ip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519.1408860897173, length=303 2014-08-24 06:15:44,068 INFO [RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0] wal.HLogSplitter(274): DistributedLogReplay = false 2014-08-24 06:15:44,069 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519-splitting%2Fip-10-229-0-82.us-west-2.compute.internal%252C57246%252C1408860896519.1408860897173 2014-08-24 06:15:44,069 DEBUG [Thread-434-EventThread] zookeeper.ZooKeeperWatcher(312): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519-splitting%2Fip-10-229-0-82.us-west-2.compute.internal%252C57246%252C1408860896519.1408860897173 2014-08-24 06:15:44,070 INFO [RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0] util.FSHDFSUtils(173): Recovering lease on dfs file hdfs://localhost:59365/user/apurtell/hbase/WALs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519-splitting/ip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519.1408860897173 2014-08-24 06:15:44,074 INFO [RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0] util.FSHDFSUtils(260): recoverLease=true, attempt=0 on file=hdfs://localhost:59365/user/apurtell/hbase/WALs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519-splitting/ip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519.1408860897173 after 4ms 2014-08-24 06:15:44,079 DEBUG [RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0-Writer-0] wal.HLogSplitter$WriterThread(830): Writer thread Thread[RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0-Writer-0,5,main]: starting 2014-08-24 06:15:44,079 DEBUG [RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0-Writer-1] wal.HLogSplitter$WriterThread(830): Writer thread Thread[RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0-Writer-1,5,main]: starting 2014-08-24 06:15:44,080 DEBUG [RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0-Writer-2] wal.HLogSplitter$WriterThread(830): Writer thread Thread[RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0-Writer-2,5,main]: starting 2014-08-24 06:15:44,086 DEBUG [RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0] wal.HLogSplitter(359): Finishing writing output logs and closing down. 2014-08-24 06:15:44,086 INFO [RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0] wal.HLogSplitter$OutputSink(956): Waiting for split writer threads to finish 2014-08-24 06:15:44,092 DEBUG [RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0-Writer-2] wal.HLogSplitter$LogRecoveredEditsOutputSink(1233): Creating writer path=hdfs://localhost:59365/user/apurtell/hbase/data/hbase/namespace/43a5af54b31ca15a1e49e35d4ace9f79/recovered.edits/0000000000000000002.temp region=43a5af54b31ca15a1e49e35d4ace9f79 2014-08-24 06:15:44,092 INFO [RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0] wal.HLogSplitter$OutputSink(974): Split writers finished 2014-08-24 06:15:44,093 DEBUG [RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0] wal.HLogSplitter$LogRecoveredEditsOutputSink(1063): Submitting close of hdfs://localhost:59365/user/apurtell/hbase/data/hbase/namespace/43a5af54b31ca15a1e49e35d4ace9f79/recovered.edits/0000000000000000002.temp 2014-08-24 06:15:44,093 DEBUG [split-log-closeStream-1] wal.HLogSplitter$LogRecoveredEditsOutputSink$2(1067): Closing hdfs://localhost:59365/user/apurtell/hbase/data/hbase/namespace/43a5af54b31ca15a1e49e35d4ace9f79/recovered.edits/0000000000000000002.temp 2014-08-24 06:15:44,102 INFO [IPC Server handler 9 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741870_1046{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:44,102 INFO [IPC Server handler 2 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741870_1046{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:44,103 INFO [IPC Server handler 3 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741870_1046 size 303 2014-08-24 06:15:44,103 INFO [split-log-closeStream-1] wal.HLogSplitter$LogRecoveredEditsOutputSink$2(1075): Closed wap hdfs://localhost:59365/user/apurtell/hbase/data/hbase/namespace/43a5af54b31ca15a1e49e35d4ace9f79/recovered.edits/0000000000000000002.temp (wrote 2 edits in 6ms) 2014-08-24 06:15:44,105 DEBUG [split-log-closeStream-1] wal.HLogSplitter$LogRecoveredEditsOutputSink$2(1106): Rename hdfs://localhost:59365/user/apurtell/hbase/data/hbase/namespace/43a5af54b31ca15a1e49e35d4ace9f79/recovered.edits/0000000000000000002.temp to hdfs://localhost:59365/user/apurtell/hbase/data/hbase/namespace/43a5af54b31ca15a1e49e35d4ace9f79/recovered.edits/0000000000000000003 2014-08-24 06:15:44,105 INFO [RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0] wal.HLogSplitter(372): Processed 2 edits across 1 regions; log file=hdfs://localhost:59365/user/apurtell/hbase/WALs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519-splitting/ip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519.1408860897173 is corrupted = false progress failed = false 2014-08-24 06:15:44,107 DEBUG [Thread-434-EventThread] zookeeper.ZooKeeperWatcher(312): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519-splitting%2Fip-10-229-0-82.us-west-2.compute.internal%252C57246%252C1408860896519.1408860897173 2014-08-24 06:15:44,107 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519-splitting%2Fip-10-229-0-82.us-west-2.compute.internal%252C57246%252C1408860896519.1408860897173 2014-08-24 06:15:44,107 INFO [RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0] handler.HLogSplitterHandler(126): successfully transitioned task /hbase/splitWAL/WALs%2Fip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519-splitting%2Fip-10-229-0-82.us-west-2.compute.internal%252C57246%252C1408860896519.1408860897173 to final state DONE ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,107 INFO [RS_LOG_REPLAY_OPS-ip-10-229-0-82:55743-0] handler.HLogSplitterHandler(110): worker ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 done with task /hbase/splitWAL/WALs%2Fip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519-splitting%2Fip-10-229-0-82.us-west-2.compute.internal%252C57246%252C1408860896519.1408860897173 in 59ms 2014-08-24 06:15:44,107 INFO [Thread-434-EventThread] master.SplitLogManager(767): task /hbase/splitWAL/WALs%2Fip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519-splitting%2Fip-10-229-0-82.us-west-2.compute.internal%252C57246%252C1408860896519.1408860897173 entered state: DONE ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,111 DEBUG [Thread-434-EventThread] wal.HLogSplitter(462): Archived processed log hdfs://localhost:59365/user/apurtell/hbase/WALs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519-splitting/ip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519.1408860897173 to hdfs://localhost:59365/user/apurtell/hbase/oldWALs/ip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519.1408860897173 2014-08-24 06:15:44,112 INFO [Thread-434-EventThread] master.SplitLogManager(681): Done splitting /hbase/splitWAL/WALs%2Fip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519-splitting%2Fip-10-229-0-82.us-west-2.compute.internal%252C57246%252C1408860896519.1408860897173 2014-08-24 06:15:44,113 INFO [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.SplitLogManager(401): finished splitting (more than or equal to) 303 bytes in 1 log files in [hdfs://localhost:59365/user/apurtell/hbase/WALs/ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519-splitting] in 75ms 2014-08-24 06:15:44,113 DEBUG [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.RegionStates(392): Adding to processed servers ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:44,113 DEBUG [Thread-434-EventThread] zookeeper.ZooKeeperWatcher(312): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519-splitting%2Fip-10-229-0-82.us-west-2.compute.internal%252C57246%252C1408860896519.1408860897173 2014-08-24 06:15:44,114 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519-splitting%2Fip-10-229-0-82.us-west-2.compute.internal%252C57246%252C1408860896519.1408860897173 2014-08-24 06:15:44,114 DEBUG [Thread-434-EventThread] master.SplitLogManager$DeleteAsyncCallback(1726): deleted /hbase/splitWAL/WALs%2Fip-10-229-0-82.us-west-2.compute.internal%2C57246%2C1408860896519-splitting%2Fip-10-229-0-82.us-west-2.compute.internal%252C57246%252C1408860896519.1408860897173 2014-08-24 06:15:44,114 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:58042-0x14806a73b1d0002, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2014-08-24 06:15:44,114 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2014-08-24 06:15:44,114 INFO [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.RegionStates(802): Transition {43a5af54b31ca15a1e49e35d4ace9f79 state=OPEN, ts=1408860901403, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} to {43a5af54b31ca15a1e49e35d4ace9f79 state=OFFLINE, ts=1408860944114, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} 2014-08-24 06:15:44,114 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] regionserver.SplitLogWorker(591): tasks arrived or departed 2014-08-24 06:15:44,114 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] regionserver.SplitLogWorker(591): tasks arrived or departed 2014-08-24 06:15:44,114 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2014-08-24 06:15:44,114 INFO [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.RegionStateStore(183): Updating row hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79. with state=OFFLINE 2014-08-24 06:15:44,114 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] regionserver.SplitLogWorker(591): tasks arrived or departed 2014-08-24 06:15:44,117 INFO [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.RegionStates(455): Offlined 43a5af54b31ca15a1e49e35d4ace9f79 from ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:44,117 INFO [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.RegionStates(507): Transitioning {e4055a2384e5839468e67fd9cd4bd3a0 state=FAILED_CLOSE, ts=1408860944021, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} will be handled by SSH for ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:44,117 INFO [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] handler.ServerShutdownHandler(215): Reassigning 2 region(s) that ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 was carrying (and 0 regions(s) that were opening on this server) 2014-08-24 06:15:44,117 INFO [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] handler.ServerShutdownHandler(251): Reassigning region with rs = {e4055a2384e5839468e67fd9cd4bd3a0 state=FAILED_CLOSE, ts=1408860944021, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} and deleting zk node if exists 2014-08-24 06:15:44,118 WARN [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] zookeeper.RecoverableZooKeeper(166): Node /hbase/region-in-transition/e4055a2384e5839468e67fd9cd4bd3a0 already deleted, retry=false 2014-08-24 06:15:44,118 INFO [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.RegionStates(802): Transition {e4055a2384e5839468e67fd9cd4bd3a0 state=FAILED_CLOSE, ts=1408860944021, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} to {e4055a2384e5839468e67fd9cd4bd3a0 state=OFFLINE, ts=1408860944118, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} 2014-08-24 06:15:44,119 INFO [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. with state=OFFLINE 2014-08-24 06:15:44,120 INFO [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.AssignmentManager(2696): Bulk assigning 2 region(s) across 3 server(s), round-robin=true 2014-08-24 06:15:44,121 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.AssignmentManager(1597): Assigning 1 region(s) to ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:44,121 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.AssignmentManager(1597): Assigning 1 region(s) to ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,121 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStates(802): Transition {43a5af54b31ca15a1e49e35d4ace9f79 state=OFFLINE, ts=1408860944114, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} to {43a5af54b31ca15a1e49e35d4ace9f79 state=PENDING_OPEN, ts=1408860944121, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:44,121 DEBUG [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.GeneralBulkAssigner(177): Timeout-on-RIT=151000 2014-08-24 06:15:44,121 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStateStore(183): Updating row hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:44,121 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.RegionStates(802): Transition {e4055a2384e5839468e67fd9cd4bd3a0 state=OFFLINE, ts=1408860944118, server=ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519} to {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_OPEN, ts=1408860944121, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:44,122 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,124 INFO [PriorityRpcServer.handler=3,queue=0,port=51857] regionserver.HRegionServer(3718): Open hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79. 2014-08-24 06:15:44,127 INFO [PriorityRpcServer.handler=6,queue=0,port=55743] regionserver.HRegionServer(3718): Open testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:44,129 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.AssignmentManager(1777): Bulk assigning done for ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:44,129 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(4470): Opening region: {ENCODED => 43a5af54b31ca15a1e49e35d4ace9f79, NAME => 'hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79.', STARTKEY => '', ENDKEY => ''} 2014-08-24 06:15:44,130 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-1] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:44,130 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table namespace 43a5af54b31ca15a1e49e35d4ace9f79 2014-08-24 06:15:44,130 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(639): Instantiated hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79. 2014-08-24 06:15:44,131 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(4470): Opening region: {ENCODED => e4055a2384e5839468e67fd9cd4bd3a0, NAME => 'testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0.', STARTKEY => 'A', ENDKEY => 'Z'} 2014-08-24 06:15:44,131 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.AssignmentManager(1777): Bulk assigning done for ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,131 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-0] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:44,131 DEBUG [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.GeneralBulkAssigner(153): bulk assigning total 2 regions to 3 servers, took 9ms, with 2 regions still in transition 2014-08-24 06:15:44,131 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testAssignRacingWithSSH e4055a2384e5839468e67fd9cd4bd3a0 2014-08-24 06:15:44,131 INFO [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.AssignmentManager(2703): Bulk assigning done 2014-08-24 06:15:44,132 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(639): Instantiated testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:44,132 DEBUG [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] master.DeadServer(118): Finished processing ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:44,132 INFO [MASTER_SERVER_OPERATIONS-ip-10-229-0-82:57826-1] handler.ServerShutdownHandler(317): Finished processing of shutdown of ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:44,133 INFO [StoreOpener-43a5af54b31ca15a1e49e35d4ace9f79-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:44,134 DEBUG [StoreOpener-43a5af54b31ca15a1e49e35d4ace9f79-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/hbase/namespace/43a5af54b31ca15a1e49e35d4ace9f79/info 2014-08-24 06:15:44,134 INFO [StoreOpener-e4055a2384e5839468e67fd9cd4bd3a0-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:44,135 DEBUG [StoreOpener-e4055a2384e5839468e67fd9cd4bd3a0-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testAssignRacingWithSSH/e4055a2384e5839468e67fd9cd4bd3a0/FAMILY 2014-08-24 06:15:44,136 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testAssignRacingWithSSH/e4055a2384e5839468e67fd9cd4bd3a0 2014-08-24 06:15:44,136 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(3107): Found 1 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/hbase/namespace/43a5af54b31ca15a1e49e35d4ace9f79 2014-08-24 06:15:44,137 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(740): Onlined e4055a2384e5839468e67fd9cd4bd3a0; next sequenceid=1 2014-08-24 06:15:44,137 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(3190): Replaying edits from hdfs://localhost:59365/user/apurtell/hbase/data/hbase/namespace/43a5af54b31ca15a1e49e35d4ace9f79/recovered.edits/0000000000000000003 2014-08-24 06:15:44,137 INFO [PostOpenDeployTasks:e4055a2384e5839468e67fd9cd4bd3a0] regionserver.HRegionServer(1796): Post open deploy tasks for region=testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:44,138 DEBUG [FifoRpcScheduler.handler7-thread-5] master.AssignmentManager(4068): Got transition OPENED for {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_OPEN, ts=1408860944121, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,138 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(802): Transition {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_OPEN, ts=1408860944121, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {e4055a2384e5839468e67fd9cd4bd3a0 state=OPEN, ts=1408860944138, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:44,138 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,140 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(359): Onlined e4055a2384e5839468e67fd9cd4bd3a0 on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,140 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(367): Offlined e4055a2384e5839468e67fd9cd4bd3a0 from ip-10-229-0-82.us-west-2.compute.internal,57246,1408860896519 2014-08-24 06:15:44,141 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(3339): Applied 2, skipped 0, firstSequenceidInLog=2, maxSequenceidInLog=3, path=hdfs://localhost:59365/user/apurtell/hbase/data/hbase/namespace/43a5af54b31ca15a1e49e35d4ace9f79/recovered.edits/0000000000000000003 2014-08-24 06:15:44,141 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(1689): Started memstore flush for hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79., current region memstore size 344; wal is null, using passed sequenceid=3 2014-08-24 06:15:44,143 INFO [PostOpenDeployTasks:e4055a2384e5839468e67fd9cd4bd3a0] regionserver.HRegionServer(1827): Finished post open deploy task for testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:44,143 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-0] handler.OpenRegionHandler(185): Opened testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,153 INFO [Thread-670] client.HBaseAdmin$5(917): Started disable of testAssignRacingWithSSH 2014-08-24 06:15:44,154 INFO [FifoRpcScheduler.handler7-thread-2] master.HMaster(1979): Client=apurtell//10.229.0.82 disable testAssignRacingWithSSH 2014-08-24 06:15:44,156 DEBUG [FifoRpcScheduler.handler7-thread-2] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testAssignRacingWithSSH/write-master:578260000000001 2014-08-24 06:15:44,163 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(130): Attempting to disable table testAssignRacingWithSSH 2014-08-24 06:15:44,164 INFO [Thread-670] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:44,164 DEBUG [Thread-670] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@6b61dd06 2014-08-24 06:15:44,167 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(174): Offlining 2 regions. 2014-08-24 06:15:44,167 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2420): Starting unassign of testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. (offlining), current state: {194f89dd1ff2fa120355638b0291f840 state=OPEN, ts=1408860905613, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:44,168 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.AssignmentManager(2420): Starting unassign of testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. (offlining), current state: {e4055a2384e5839468e67fd9cd4bd3a0 state=OPEN, ts=1408860944138, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:44,168 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(802): Transition {194f89dd1ff2fa120355638b0291f840 state=OPEN, ts=1408860905613, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {194f89dd1ff2fa120355638b0291f840 state=PENDING_CLOSE, ts=1408860944168, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:44,168 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.RegionStates(802): Transition {e4055a2384e5839468e67fd9cd4bd3a0 state=OPEN, ts=1408860944138, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_CLOSE, ts=1408860944168, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:44,168 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. with state=PENDING_CLOSE 2014-08-24 06:15:44,168 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. with state=PENDING_CLOSE 2014-08-24 06:15:44,168 DEBUG [Thread-670-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:44,168 DEBUG [Thread-670] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:44,169 DEBUG [Thread-670-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d0036 connected 2014-08-24 06:15:44,171 INFO [PriorityRpcServer.handler=0,queue=0,port=58042] regionserver.HRegionServer(3859): Close 194f89dd1ff2fa120355638b0291f840, via zk=no, znode version=-1, on null 2014-08-24 06:15:44,171 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] handler.CloseRegionHandler(128): Processing close of testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. 2014-08-24 06:15:44,171 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 for region testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. 2014-08-24 06:15:44,172 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(1109): Closing testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840.: disabling compactions & flushes 2014-08-24 06:15:44,172 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(1136): Updates disabled for region testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. 2014-08-24 06:15:44,173 INFO [StoreCloserThread-testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:44,173 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:44,173 INFO [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(1218): Closed testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. 2014-08-24 06:15:44,174 INFO [PriorityRpcServer.handler=9,queue=0,port=55743] regionserver.HRegionServer(3859): Close e4055a2384e5839468e67fd9cd4bd3a0, via zk=no, znode version=-1, on null 2014-08-24 06:15:44,174 DEBUG [FifoRpcScheduler.handler7-thread-4] master.AssignmentManager(4068): Got transition CLOSED for {194f89dd1ff2fa120355638b0291f840 state=PENDING_CLOSE, ts=1408860944168, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,175 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] handler.CloseRegionHandler(128): Processing close of testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:44,175 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 for region testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:44,175 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(1109): Closing testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0.: disabling compactions & flushes 2014-08-24 06:15:44,175 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(1136): Updates disabled for region testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:44,176 INFO [StoreCloserThread-testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:44,176 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:44,176 INFO [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(1218): Closed testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:44,176 DEBUG [FifoRpcScheduler.handler7-thread-1] master.AssignmentManager(4068): Got transition CLOSED for {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_CLOSE, ts=1408860944168, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,179 DEBUG [Thread-670] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@6b61dd06 2014-08-24 06:15:44,179 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(802): Transition {194f89dd1ff2fa120355638b0291f840 state=PENDING_CLOSE, ts=1408860944168, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {194f89dd1ff2fa120355638b0291f840 state=OFFLINE, ts=1408860944179, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:44,179 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. with state=OFFLINE 2014-08-24 06:15:44,180 DEBUG [Thread-670] client.HBaseAdmin(956): Sleeping= 100ms, waiting for all regions to be disabled in testAssignRacingWithSSH 2014-08-24 06:15:44,180 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741871_1047{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW]]} size 0 2014-08-24 06:15:44,182 INFO [IPC Server handler 5 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741871_1047{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW]]} size 0 2014-08-24 06:15:44,182 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741871_1047 size 1045 2014-08-24 06:15:44,182 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(802): Transition {e4055a2384e5839468e67fd9cd4bd3a0 state=PENDING_CLOSE, ts=1408860944168, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {e4055a2384e5839468e67fd9cd4bd3a0 state=OFFLINE, ts=1408860944182, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:44,182 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStateStore(183): Updating row testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. with state=OFFLINE 2014-08-24 06:15:44,182 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.DefaultStoreFlusher(82): Flushed, sequenceid=3, memsize=344, hasBloomFilter=true, into tmp file hdfs://localhost:59365/user/apurtell/hbase/data/hbase/namespace/43a5af54b31ca15a1e49e35d4ace9f79/.tmp/0c21c3d5c64f4858b3b77afcbd4c68d2 2014-08-24 06:15:44,183 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(455): Offlined 194f89dd1ff2fa120355638b0291f840 from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,183 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] handler.CloseRegionHandler(182): Closed testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. 2014-08-24 06:15:44,184 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(455): Offlined e4055a2384e5839468e67fd9cd4bd3a0 from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,185 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] handler.CloseRegionHandler(182): Closed testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. 2014-08-24 06:15:44,217 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegionFileSystem(376): Committing store file hdfs://localhost:59365/user/apurtell/hbase/data/hbase/namespace/43a5af54b31ca15a1e49e35d4ace9f79/.tmp/0c21c3d5c64f4858b3b77afcbd4c68d2 as hdfs://localhost:59365/user/apurtell/hbase/data/hbase/namespace/43a5af54b31ca15a1e49e35d4ace9f79/info/0c21c3d5c64f4858b3b77afcbd4c68d2 2014-08-24 06:15:44,218 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler$BulkDisabler(230): Disable waiting until done; 300000 ms remaining; [] 2014-08-24 06:15:44,219 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(190): Disabled table, testAssignRacingWithSSH, is done=true 2014-08-24 06:15:44,221 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testAssignRacingWithSSH/write-master:578260000000001 2014-08-24 06:15:44,224 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HStore(862): Added hdfs://localhost:59365/user/apurtell/hbase/data/hbase/namespace/43a5af54b31ca15a1e49e35d4ace9f79/info/0c21c3d5c64f4858b3b77afcbd4c68d2, entries=2, sequenceid=3, filesize=1.0 K 2014-08-24 06:15:44,224 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(1835): Finished memstore flush of ~344/344, currentsize=0/0 for region hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79. in 83ms, sequenceid=3, compaction requested=false; wal=null 2014-08-24 06:15:44,225 INFO [IPC Server handler 9 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741870_1046 127.0.0.1:60268 127.0.0.1:39258 127.0.0.1:42916 2014-08-24 06:15:44,226 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(3171): Deleted recovered.edits file=hdfs://localhost:59365/user/apurtell/hbase/data/hbase/namespace/43a5af54b31ca15a1e49e35d4ace9f79/recovered.edits/0000000000000000003 2014-08-24 06:15:44,227 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(740): Onlined 43a5af54b31ca15a1e49e35d4ace9f79; next sequenceid=4 2014-08-24 06:15:44,227 INFO [PostOpenDeployTasks:43a5af54b31ca15a1e49e35d4ace9f79] regionserver.HRegionServer(1796): Post open deploy tasks for region=hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79. 2014-08-24 06:15:44,228 DEBUG [FifoRpcScheduler.handler7-thread-5] master.AssignmentManager(4068): Got transition OPENED for {43a5af54b31ca15a1e49e35d4ace9f79 state=PENDING_OPEN, ts=1408860944121, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:44,228 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(802): Transition {43a5af54b31ca15a1e49e35d4ace9f79 state=PENDING_OPEN, ts=1408860944121, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {43a5af54b31ca15a1e49e35d4ace9f79 state=OPEN, ts=1408860944228, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:44,228 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStateStore(183): Updating row hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79. with state=OPEN&openSeqNum=4&server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:44,230 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(359): Onlined 43a5af54b31ca15a1e49e35d4ace9f79 on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:44,233 INFO [PostOpenDeployTasks:43a5af54b31ca15a1e49e35d4ace9f79] regionserver.HRegionServer(1827): Finished post open deploy task for hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79. 2014-08-24 06:15:44,233 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] handler.OpenRegionHandler(185): Opened hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79. on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:44,281 INFO [Thread-670] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:44,281 DEBUG [Thread-670] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@294b1c18 2014-08-24 06:15:44,283 DEBUG [Thread-670-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:44,283 DEBUG [Thread-670] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:44,284 DEBUG [Thread-670-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d0037 connected 2014-08-24 06:15:44,289 DEBUG [Thread-670] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@294b1c18 2014-08-24 06:15:44,290 INFO [Thread-670] client.HBaseAdmin(971): Disabled testAssignRacingWithSSH 2014-08-24 06:15:44,291 INFO [FifoRpcScheduler.handler7-thread-2] master.HMaster(1816): Client=apurtell//10.229.0.82 delete testAssignRacingWithSSH 2014-08-24 06:15:44,292 DEBUG [FifoRpcScheduler.handler7-thread-2] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testAssignRacingWithSSH/write-master:578260000000002 2014-08-24 06:15:44,300 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.TableEventHandler(123): Handling table operation C_M_DELETE_TABLE on table testAssignRacingWithSSH 2014-08-24 06:15:44,307 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(93): Deleting regions from META 2014-08-24 06:15:44,310 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] catalog.MetaEditor(495): Deleted [{ENCODED => 194f89dd1ff2fa120355638b0291f840, NAME => 'testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840.', STARTKEY => '', ENDKEY => ''}, {ENCODED => e4055a2384e5839468e67fd9cd4bd3a0, NAME => 'testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0.', STARTKEY => 'A', ENDKEY => 'Z'}] 2014-08-24 06:15:44,311 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testAssignRacingWithSSH,,1408860905560.194f89dd1ff2fa120355638b0291f840. from FS 2014-08-24 06:15:44,312 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testAssignRacingWithSSH/194f89dd1ff2fa120355638b0291f840 2014-08-24 06:15:44,314 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testAssignRacingWithSSH/194f89dd1ff2fa120355638b0291f840/FAMILY] 2014-08-24 06:15:44,317 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741868_1044 127.0.0.1:45654 127.0.0.1:39258 127.0.0.1:60268 2014-08-24 06:15:44,317 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testAssignRacingWithSSH/194f89dd1ff2fa120355638b0291f840 2014-08-24 06:15:44,317 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testAssignRacingWithSSH,A,1408860905695.e4055a2384e5839468e67fd9cd4bd3a0. from FS 2014-08-24 06:15:44,318 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testAssignRacingWithSSH/e4055a2384e5839468e67fd9cd4bd3a0 2014-08-24 06:15:44,320 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testAssignRacingWithSSH/e4055a2384e5839468e67fd9cd4bd3a0/FAMILY] 2014-08-24 06:15:44,323 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741869_1045 127.0.0.1:45654 127.0.0.1:39258 127.0.0.1:60268 2014-08-24 06:15:44,323 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testAssignRacingWithSSH/e4055a2384e5839468e67fd9cd4bd3a0 2014-08-24 06:15:44,324 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741867_1043 127.0.0.1:39258 127.0.0.1:60268 127.0.0.1:42916 2014-08-24 06:15:44,324 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(114): Table 'testAssignRacingWithSSH' archived! 2014-08-24 06:15:44,324 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(117): Removing 'testAssignRacingWithSSH' descriptor. 2014-08-24 06:15:44,325 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(121): Removing 'testAssignRacingWithSSH' from region states. 2014-08-24 06:15:44,325 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(125): Marking 'testAssignRacingWithSSH' as deleted. 2014-08-24 06:15:44,327 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testAssignRacingWithSSH/write-master:578260000000002 2014-08-24 06:15:44,404 INFO [Thread-670] client.HBaseAdmin(705): Deleted testAssignRacingWithSSH 2014-08-24 06:15:44,417 INFO [main] hbase.ResourceChecker(171): after: master.TestZKLessAMOnCluster#testAssignRacingWithSSH Thread=486 (was 549), OpenFileDescriptor=661 (was 781), MaxFileDescriptor=65536 (was 65536), SystemLoadAverage=17 (was 30), ProcessCount=152 (was 152), AvailableMemoryMB=27871 (was 27869) - AvailableMemoryMB LEAK? -, ConnectionCount=9 (was 10) 2014-08-24 06:15:44,433 INFO [main] hbase.ResourceChecker(147): before: master.TestZKLessAMOnCluster#testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState Thread=486, OpenFileDescriptor=661, MaxFileDescriptor=65536, SystemLoadAverage=17, ProcessCount=152, AvailableMemoryMB=27869, ConnectionCount=9 2014-08-24 06:15:44,435 INFO [FifoRpcScheduler.handler7-thread-1] master.HMaster(1747): Client=apurtell//10.229.0.82 create 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2014-08-24 06:15:44,439 DEBUG [FifoRpcScheduler.handler7-thread-1] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/write-master:578260000000000 2014-08-24 06:15:44,444 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.CreateTableHandler(148): Create table testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState 2014-08-24 06:15:44,451 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741872_1048{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:44,452 INFO [IPC Server handler 5 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741872_1048{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:44,452 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741872_1048 size 341 2014-08-24 06:15:44,454 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] util.FSTableDescriptors(640): Wrote descriptor into: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/.tabledesc/.tableinfo.0000000001 2014-08-24 06:15:44,455 INFO [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-1] regionserver.HRegion(4284): creating HRegion testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState HTD == 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState 2014-08-24 06:15:44,455 INFO [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-2] regionserver.HRegion(4284): creating HRegion testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState HTD == 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState 2014-08-24 06:15:44,455 INFO [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-4] regionserver.HRegion(4284): creating HRegion testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState HTD == 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState 2014-08-24 06:15:44,455 INFO [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-3] regionserver.HRegion(4284): creating HRegion testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState HTD == 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState 2014-08-24 06:15:44,456 INFO [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-5] regionserver.HRegion(4284): creating HRegion testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState HTD == 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState 2014-08-24 06:15:44,477 INFO [IPC Server handler 2 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741875_1051{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW]]} size 0 2014-08-24 06:15:44,478 INFO [IPC Server handler 1 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741873_1049{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:44,479 INFO [IPC Server handler 3 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741875_1051{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW]]} size 0 2014-08-24 06:15:44,479 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741873_1049{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:44,481 INFO [IPC Server handler 8 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741874_1050{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:44,481 DEBUG [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-4] regionserver.HRegion(639): Instantiated testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f. 2014-08-24 06:15:44,481 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741874_1050{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:44,482 DEBUG [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-4] regionserver.HRegion(1109): Closing testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f.: disabling compactions & flushes 2014-08-24 06:15:44,482 DEBUG [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-1] regionserver.HRegion(639): Instantiated testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. 2014-08-24 06:15:44,482 DEBUG [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-4] regionserver.HRegion(1136): Updates disabled for region testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f. 2014-08-24 06:15:44,482 INFO [IPC Server handler 7 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741874_1050{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:44,483 INFO [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-4] regionserver.HRegion(1218): Closed testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f. 2014-08-24 06:15:44,482 DEBUG [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-1] regionserver.HRegion(1109): Closing testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb.: disabling compactions & flushes 2014-08-24 06:15:44,483 INFO [IPC Server handler 7 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741873_1049 size 93 2014-08-24 06:15:44,483 DEBUG [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-1] regionserver.HRegion(1136): Updates disabled for region testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. 2014-08-24 06:15:44,483 INFO [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-1] regionserver.HRegion(1218): Closed testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. 2014-08-24 06:15:44,483 INFO [IPC Server handler 2 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741875_1051 size 94 2014-08-24 06:15:44,486 INFO [IPC Server handler 5 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741876_1052{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:44,486 INFO [IPC Server handler 8 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741876_1052{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:44,486 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741877_1053{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:44,486 DEBUG [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-2] regionserver.HRegion(639): Instantiated testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3. 2014-08-24 06:15:44,487 DEBUG [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-2] regionserver.HRegion(1109): Closing testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3.: disabling compactions & flushes 2014-08-24 06:15:44,487 DEBUG [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-2] regionserver.HRegion(1136): Updates disabled for region testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3. 2014-08-24 06:15:44,487 INFO [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-2] regionserver.HRegion(1218): Closed testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3. 2014-08-24 06:15:44,487 INFO [IPC Server handler 9 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741876_1052 size 94 2014-08-24 06:15:44,487 DEBUG [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-3] regionserver.HRegion(639): Instantiated testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. 2014-08-24 06:15:44,487 DEBUG [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-3] regionserver.HRegion(1109): Closing testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d.: disabling compactions & flushes 2014-08-24 06:15:44,487 INFO [IPC Server handler 1 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741877_1053 size 93 2014-08-24 06:15:44,487 DEBUG [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-3] regionserver.HRegion(1136): Updates disabled for region testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. 2014-08-24 06:15:44,487 INFO [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-3] regionserver.HRegion(1218): Closed testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. 2014-08-24 06:15:44,487 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741877_1053 size 93 2014-08-24 06:15:44,527 DEBUG [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-5] regionserver.HRegion(639): Instantiated testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. 2014-08-24 06:15:44,527 DEBUG [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-5] regionserver.HRegion(1109): Closing testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603.: disabling compactions & flushes 2014-08-24 06:15:44,527 DEBUG [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-5] regionserver.HRegion(1136): Updates disabled for region testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. 2014-08-24 06:15:44,527 INFO [RegionOpenAndInitThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState-5] regionserver.HRegion(1218): Closed testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. 2014-08-24 06:15:44,532 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] catalog.MetaEditor(278): Added 5 2014-08-24 06:15:44,532 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.AssignmentManager(2696): Bulk assigning 5 region(s) across 3 server(s), round-robin=true 2014-08-24 06:15:44,533 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.AssignmentManager(1597): Assigning 1 region(s) to ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:44,533 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.AssignmentManager(1597): Assigning 2 region(s) to ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,533 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.GeneralBulkAssigner(177): Timeout-on-RIT=152000 2014-08-24 06:15:44,533 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.RegionStates(802): Transition {4d4fec75cd98afbba2cf4df10292c3eb state=OFFLINE, ts=1408860944532, server=null} to {4d4fec75cd98afbba2cf4df10292c3eb state=PENDING_OPEN, ts=1408860944533, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:44,533 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.AssignmentManager(1597): Assigning 2 region(s) to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,533 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStates(802): Transition {e859174d1aec5977deb209db04be31d3 state=OFFLINE, ts=1408860944532, server=null} to {e859174d1aec5977deb209db04be31d3 state=PENDING_OPEN, ts=1408860944533, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:44,533 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.RegionStates(802): Transition {578ca2c80e167109c1bb4231a65aec1f state=OFFLINE, ts=1408860944532, server=null} to {578ca2c80e167109c1bb4231a65aec1f state=PENDING_OPEN, ts=1408860944533, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:44,533 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,533 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,533 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:44,535 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.RegionStates(802): Transition {39b8af357f05404077613e6bb6e5c603 state=OFFLINE, ts=1408860944532, server=null} to {39b8af357f05404077613e6bb6e5c603 state=PENDING_OPEN, ts=1408860944535, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:44,535 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,538 INFO [PriorityRpcServer.handler=1,queue=0,port=51857] regionserver.HRegionServer(3718): Open testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3. 2014-08-24 06:15:44,539 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.RegionStates(802): Transition {8229a7b7ed833545f9d1bcb8fab66f4d state=OFFLINE, ts=1408860944532, server=null} to {8229a7b7ed833545f9d1bcb8fab66f4d state=PENDING_OPEN, ts=1408860944539, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:44,540 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,541 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.AssignmentManager(1777): Bulk assigning done for ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:44,541 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(4470): Opening region: {ENCODED => e859174d1aec5977deb209db04be31d3, NAME => 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3.', STARTKEY => 'A', ENDKEY => 'I'} 2014-08-24 06:15:44,542 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-0] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:44,542 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState e859174d1aec5977deb209db04be31d3 2014-08-24 06:15:44,542 INFO [PriorityRpcServer.handler=0,queue=0,port=55743] regionserver.HRegionServer(3718): Open testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. 2014-08-24 06:15:44,542 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(639): Instantiated testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3. 2014-08-24 06:15:44,544 INFO [PriorityRpcServer.handler=1,queue=0,port=58042] regionserver.HRegionServer(3718): Open testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f. 2014-08-24 06:15:44,546 INFO [PriorityRpcServer.handler=0,queue=0,port=55743] regionserver.HRegionServer(3718): Open testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. 2014-08-24 06:15:44,546 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(4470): Opening region: {ENCODED => 4d4fec75cd98afbba2cf4df10292c3eb, NAME => 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb.', STARTKEY => '', ENDKEY => 'A'} 2014-08-24 06:15:44,546 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-1] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:44,547 INFO [StoreOpener-e859174d1aec5977deb209db04be31d3-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:44,547 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(4470): Opening region: {ENCODED => 39b8af357f05404077613e6bb6e5c603, NAME => 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603.', STARTKEY => 'Z', ENDKEY => ''} 2014-08-24 06:15:44,547 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.AssignmentManager(1777): Bulk assigning done for ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,547 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState 4d4fec75cd98afbba2cf4df10292c3eb 2014-08-24 06:15:44,547 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-2] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:44,547 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(639): Instantiated testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. 2014-08-24 06:15:44,548 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState 39b8af357f05404077613e6bb6e5c603 2014-08-24 06:15:44,548 DEBUG [StoreOpener-e859174d1aec5977deb209db04be31d3-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/e859174d1aec5977deb209db04be31d3/FAMILY 2014-08-24 06:15:44,548 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(639): Instantiated testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. 2014-08-24 06:15:44,548 INFO [PriorityRpcServer.handler=1,queue=0,port=58042] regionserver.HRegionServer(3718): Open testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. 2014-08-24 06:15:44,549 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/e859174d1aec5977deb209db04be31d3 2014-08-24 06:15:44,549 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(4470): Opening region: {ENCODED => 578ca2c80e167109c1bb4231a65aec1f, NAME => 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f.', STARTKEY => 'Q', ENDKEY => 'Z'} 2014-08-24 06:15:44,549 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-0] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:44,549 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState 578ca2c80e167109c1bb4231a65aec1f 2014-08-24 06:15:44,549 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.AssignmentManager(1777): Bulk assigning done for ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,549 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(4470): Opening region: {ENCODED => 8229a7b7ed833545f9d1bcb8fab66f4d, NAME => 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d.', STARTKEY => 'I', ENDKEY => 'Q'} 2014-08-24 06:15:44,549 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(639): Instantiated testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f. 2014-08-24 06:15:44,550 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.GeneralBulkAssigner(153): bulk assigning total 5 regions to 3 servers, took 17ms, with 5 regions still in transition 2014-08-24 06:15:44,550 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.AssignmentManager(2703): Bulk assigning done 2014-08-24 06:15:44,550 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-1] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:44,550 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(740): Onlined e859174d1aec5977deb209db04be31d3; next sequenceid=1 2014-08-24 06:15:44,550 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState 8229a7b7ed833545f9d1bcb8fab66f4d 2014-08-24 06:15:44,550 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(639): Instantiated testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. 2014-08-24 06:15:44,550 INFO [PostOpenDeployTasks:e859174d1aec5977deb209db04be31d3] regionserver.HRegionServer(1796): Post open deploy tasks for region=testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3. 2014-08-24 06:15:44,551 DEBUG [FifoRpcScheduler.handler7-thread-5] master.AssignmentManager(4068): Got transition OPENED for {e859174d1aec5977deb209db04be31d3 state=PENDING_OPEN, ts=1408860944533, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:44,551 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(802): Transition {e859174d1aec5977deb209db04be31d3 state=PENDING_OPEN, ts=1408860944533, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {e859174d1aec5977deb209db04be31d3 state=OPEN, ts=1408860944551, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:44,551 INFO [StoreOpener-4d4fec75cd98afbba2cf4df10292c3eb-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:44,552 INFO [StoreOpener-39b8af357f05404077613e6bb6e5c603-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:44,551 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:44,553 DEBUG [StoreOpener-4d4fec75cd98afbba2cf4df10292c3eb-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/4d4fec75cd98afbba2cf4df10292c3eb/FAMILY 2014-08-24 06:15:44,553 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/write-master:578260000000000 2014-08-24 06:15:44,553 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.CreateTableHandler(173): failed. null 2014-08-24 06:15:44,553 DEBUG [StoreOpener-39b8af357f05404077613e6bb6e5c603-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/39b8af357f05404077613e6bb6e5c603/FAMILY 2014-08-24 06:15:44,554 INFO [StoreOpener-578ca2c80e167109c1bb4231a65aec1f-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:44,554 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/4d4fec75cd98afbba2cf4df10292c3eb 2014-08-24 06:15:44,554 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/39b8af357f05404077613e6bb6e5c603 2014-08-24 06:15:44,555 DEBUG [StoreOpener-578ca2c80e167109c1bb4231a65aec1f-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/578ca2c80e167109c1bb4231a65aec1f/FAMILY 2014-08-24 06:15:44,555 INFO [StoreOpener-8229a7b7ed833545f9d1bcb8fab66f4d-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:44,556 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(359): Onlined e859174d1aec5977deb209db04be31d3 on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:44,556 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/578ca2c80e167109c1bb4231a65aec1f 2014-08-24 06:15:44,556 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(740): Onlined 4d4fec75cd98afbba2cf4df10292c3eb; next sequenceid=1 2014-08-24 06:15:44,556 DEBUG [StoreOpener-8229a7b7ed833545f9d1bcb8fab66f4d-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/8229a7b7ed833545f9d1bcb8fab66f4d/FAMILY 2014-08-24 06:15:44,556 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(740): Onlined 39b8af357f05404077613e6bb6e5c603; next sequenceid=1 2014-08-24 06:15:44,556 INFO [PostOpenDeployTasks:4d4fec75cd98afbba2cf4df10292c3eb] regionserver.HRegionServer(1796): Post open deploy tasks for region=testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. 2014-08-24 06:15:44,556 INFO [PostOpenDeployTasks:39b8af357f05404077613e6bb6e5c603] regionserver.HRegionServer(1796): Post open deploy tasks for region=testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. 2014-08-24 06:15:44,557 DEBUG [FifoRpcScheduler.handler7-thread-3] master.AssignmentManager(4068): Got transition OPENED for {4d4fec75cd98afbba2cf4df10292c3eb state=PENDING_OPEN, ts=1408860944533, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,557 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/8229a7b7ed833545f9d1bcb8fab66f4d 2014-08-24 06:15:44,557 DEBUG [FifoRpcScheduler.handler7-thread-2] master.AssignmentManager(4068): Got transition OPENED for {39b8af357f05404077613e6bb6e5c603 state=PENDING_OPEN, ts=1408860944535, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,557 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(802): Transition {4d4fec75cd98afbba2cf4df10292c3eb state=PENDING_OPEN, ts=1408860944533, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {4d4fec75cd98afbba2cf4df10292c3eb state=OPEN, ts=1408860944557, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:44,557 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(802): Transition {39b8af357f05404077613e6bb6e5c603 state=PENDING_OPEN, ts=1408860944535, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {39b8af357f05404077613e6bb6e5c603 state=OPEN, ts=1408860944557, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:44,557 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(740): Onlined 578ca2c80e167109c1bb4231a65aec1f; next sequenceid=1 2014-08-24 06:15:44,558 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,557 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,558 INFO [PostOpenDeployTasks:578ca2c80e167109c1bb4231a65aec1f] regionserver.HRegionServer(1796): Post open deploy tasks for region=testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f. 2014-08-24 06:15:44,558 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(740): Onlined 8229a7b7ed833545f9d1bcb8fab66f4d; next sequenceid=1 2014-08-24 06:15:44,558 DEBUG [FifoRpcScheduler.handler7-thread-4] master.AssignmentManager(4068): Got transition OPENED for {578ca2c80e167109c1bb4231a65aec1f state=PENDING_OPEN, ts=1408860944533, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,559 INFO [PostOpenDeployTasks:8229a7b7ed833545f9d1bcb8fab66f4d] regionserver.HRegionServer(1796): Post open deploy tasks for region=testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. 2014-08-24 06:15:44,559 INFO [PostOpenDeployTasks:e859174d1aec5977deb209db04be31d3] regionserver.HRegionServer(1827): Finished post open deploy task for testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3. 2014-08-24 06:15:44,559 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(802): Transition {578ca2c80e167109c1bb4231a65aec1f state=PENDING_OPEN, ts=1408860944533, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {578ca2c80e167109c1bb4231a65aec1f state=OPEN, ts=1408860944559, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:44,559 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-0] handler.OpenRegionHandler(185): Opened testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3. on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:44,559 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,559 DEBUG [FifoRpcScheduler.handler7-thread-1] master.AssignmentManager(4068): Got transition OPENED for {8229a7b7ed833545f9d1bcb8fab66f4d state=PENDING_OPEN, ts=1408860944539, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,559 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(802): Transition {8229a7b7ed833545f9d1bcb8fab66f4d state=PENDING_OPEN, ts=1408860944539, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {8229a7b7ed833545f9d1bcb8fab66f4d state=OPEN, ts=1408860944559, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:44,559 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,560 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(359): Onlined 39b8af357f05404077613e6bb6e5c603 on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,562 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(359): Onlined 8229a7b7ed833545f9d1bcb8fab66f4d on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,562 INFO [PostOpenDeployTasks:39b8af357f05404077613e6bb6e5c603] regionserver.HRegionServer(1827): Finished post open deploy task for testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. 2014-08-24 06:15:44,562 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-2] handler.OpenRegionHandler(185): Opened testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,564 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(359): Onlined 578ca2c80e167109c1bb4231a65aec1f on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,564 INFO [PostOpenDeployTasks:8229a7b7ed833545f9d1bcb8fab66f4d] regionserver.HRegionServer(1827): Finished post open deploy task for testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. 2014-08-24 06:15:44,564 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] handler.OpenRegionHandler(185): Opened testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,566 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(359): Onlined 4d4fec75cd98afbba2cf4df10292c3eb on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,566 INFO [PostOpenDeployTasks:578ca2c80e167109c1bb4231a65aec1f] regionserver.HRegionServer(1827): Finished post open deploy task for testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f. 2014-08-24 06:15:44,567 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-0] handler.OpenRegionHandler(185): Opened testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f. on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,568 INFO [PostOpenDeployTasks:4d4fec75cd98afbba2cf4df10292c3eb] regionserver.HRegionServer(1827): Finished post open deploy task for testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. 2014-08-24 06:15:44,568 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-1] handler.OpenRegionHandler(185): Opened testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,753 INFO [Thread-771] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:44,754 DEBUG [Thread-771] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@662921ce 2014-08-24 06:15:44,755 DEBUG [Thread-771-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:44,755 DEBUG [Thread-771] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:44,756 DEBUG [Thread-771-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d0038 connected 2014-08-24 06:15:44,758 DEBUG [Thread-771] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@662921ce 2014-08-24 06:15:44,759 INFO [Thread-771] master.RegionStates(802): Transition {8229a7b7ed833545f9d1bcb8fab66f4d state=OPEN, ts=1408860944559, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {8229a7b7ed833545f9d1bcb8fab66f4d state=OFFLINE, ts=1408860944759, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:44,760 INFO [Thread-771] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. with state=OFFLINE 2014-08-24 06:15:44,762 INFO [Thread-771] master.RegionStates(455): Offlined 8229a7b7ed833545f9d1bcb8fab66f4d from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,762 INFO [Thread-771] master.RegionStates(802): Transition {8229a7b7ed833545f9d1bcb8fab66f4d state=OFFLINE, ts=1408860944759, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {8229a7b7ed833545f9d1bcb8fab66f4d state=PENDING_OPEN, ts=1408860944762, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:44,762 INFO [Thread-771] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,765 INFO [Thread-771] master.RegionStates(802): Transition {4d4fec75cd98afbba2cf4df10292c3eb state=OPEN, ts=1408860944557, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {4d4fec75cd98afbba2cf4df10292c3eb state=OFFLINE, ts=1408860944765, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:44,765 INFO [Thread-771] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. with state=OFFLINE 2014-08-24 06:15:44,767 INFO [Thread-771] master.RegionStates(455): Offlined 4d4fec75cd98afbba2cf4df10292c3eb from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,767 INFO [Thread-771] master.RegionStates(802): Transition {39b8af357f05404077613e6bb6e5c603 state=OPEN, ts=1408860944557, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {39b8af357f05404077613e6bb6e5c603 state=OFFLINE, ts=1408860944767, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:44,767 INFO [Thread-771] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. with state=OFFLINE 2014-08-24 06:15:44,769 INFO [Thread-771] master.RegionStates(455): Offlined 39b8af357f05404077613e6bb6e5c603 from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,769 INFO [Thread-771] master.RegionStates(517): Found region in {8229a7b7ed833545f9d1bcb8fab66f4d state=PENDING_OPEN, ts=1408860944762, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to be reassigned by SSH for ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:44,770 WARN [Thread-771] zookeeper.RecoverableZooKeeper(166): Node /hbase/region-in-transition/8229a7b7ed833545f9d1bcb8fab66f4d already deleted, retry=false 2014-08-24 06:15:44,771 INFO [Thread-771] master.RegionStates(802): Transition {8229a7b7ed833545f9d1bcb8fab66f4d state=PENDING_OPEN, ts=1408860944762, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {8229a7b7ed833545f9d1bcb8fab66f4d state=OFFLINE, ts=1408860944771, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:44,771 INFO [Thread-771] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. with state=OFFLINE 2014-08-24 06:15:44,773 INFO [Thread-771] master.RegionStates(802): Transition {8229a7b7ed833545f9d1bcb8fab66f4d state=OFFLINE, ts=1408860944771, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {8229a7b7ed833545f9d1bcb8fab66f4d state=OPEN, ts=1408860944773, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:44,773 INFO [Thread-771] master.RegionStateStore(183): Updating row testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. with state=OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,775 INFO [Thread-771] master.RegionStates(359): Onlined 8229a7b7ed833545f9d1bcb8fab66f4d on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:44,776 INFO [Thread-771] client.HBaseAdmin$5(917): Started disable of testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState 2014-08-24 06:15:44,777 INFO [FifoRpcScheduler.handler7-thread-2] master.HMaster(1979): Client=apurtell//10.229.0.82 disable testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState 2014-08-24 06:15:44,778 DEBUG [FifoRpcScheduler.handler7-thread-2] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/write-master:578260000000001 2014-08-24 06:15:44,781 INFO [FifoRpcScheduler.handler7-thread-2] handler.DisableTableHandler(99): Table testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState isn't enabled; skipping disable 2014-08-24 06:15:44,782 DEBUG [FifoRpcScheduler.handler7-thread-2] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/write-master:578260000000001 2014-08-24 06:15:44,783 DEBUG [Thread-771] hbase.HBaseTestingUtility(1493): Table: testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState already disabled, so just deleting it. 2014-08-24 06:15:44,783 INFO [FifoRpcScheduler.handler7-thread-4] master.HMaster(1816): Client=apurtell//10.229.0.82 delete testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState 2014-08-24 06:15:44,785 DEBUG [FifoRpcScheduler.handler7-thread-4] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/write-master:578260000000002 2014-08-24 06:15:44,791 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.TableEventHandler(123): Handling table operation C_M_DELETE_TABLE on table testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState 2014-08-24 06:15:44,794 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(93): Deleting regions from META 2014-08-24 06:15:44,797 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] catalog.MetaEditor(495): Deleted [{ENCODED => 4d4fec75cd98afbba2cf4df10292c3eb, NAME => 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb.', STARTKEY => '', ENDKEY => 'A'}, {ENCODED => e859174d1aec5977deb209db04be31d3, NAME => 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3.', STARTKEY => 'A', ENDKEY => 'I'}, {ENCODED => 8229a7b7ed833545f9d1bcb8fab66f4d, NAME => 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d.', STARTKEY => 'I', ENDKEY => 'Q'}, {ENCODED => 578ca2c80e167109c1bb4231a65aec1f, NAME => 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f.', STARTKEY => 'Q', ENDKEY => 'Z'}, {ENCODED => 39b8af357f05404077613e6bb6e5c603, NAME => 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603.', STARTKEY => 'Z', ENDKEY => ''}] 2014-08-24 06:15:44,799 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. from FS 2014-08-24 06:15:44,799 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/4d4fec75cd98afbba2cf4df10292c3eb 2014-08-24 06:15:44,801 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/4d4fec75cd98afbba2cf4df10292c3eb/FAMILY] 2014-08-24 06:15:44,804 INFO [IPC Server handler 3 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741873_1049 127.0.0.1:60268 127.0.0.1:42916 127.0.0.1:39258 2014-08-24 06:15:44,805 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/4d4fec75cd98afbba2cf4df10292c3eb 2014-08-24 06:15:44,805 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3. from FS 2014-08-24 06:15:44,805 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/e859174d1aec5977deb209db04be31d3 2014-08-24 06:15:44,806 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/e859174d1aec5977deb209db04be31d3/FAMILY] 2014-08-24 06:15:44,809 INFO [IPC Server handler 1 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741874_1050 127.0.0.1:39258 127.0.0.1:42916 127.0.0.1:45654 2014-08-24 06:15:44,809 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/e859174d1aec5977deb209db04be31d3 2014-08-24 06:15:44,810 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. from FS 2014-08-24 06:15:44,810 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/8229a7b7ed833545f9d1bcb8fab66f4d 2014-08-24 06:15:44,811 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/8229a7b7ed833545f9d1bcb8fab66f4d/FAMILY] 2014-08-24 06:15:44,814 INFO [IPC Server handler 2 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741876_1052 127.0.0.1:60268 127.0.0.1:39258 127.0.0.1:42916 2014-08-24 06:15:44,814 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/8229a7b7ed833545f9d1bcb8fab66f4d 2014-08-24 06:15:44,814 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f. from FS 2014-08-24 06:15:44,815 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/578ca2c80e167109c1bb4231a65aec1f 2014-08-24 06:15:44,816 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/578ca2c80e167109c1bb4231a65aec1f/FAMILY] 2014-08-24 06:15:44,819 INFO [IPC Server handler 9 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741875_1051 127.0.0.1:39258 127.0.0.1:45654 127.0.0.1:60268 2014-08-24 06:15:44,819 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/578ca2c80e167109c1bb4231a65aec1f 2014-08-24 06:15:44,819 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. from FS 2014-08-24 06:15:44,819 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/39b8af357f05404077613e6bb6e5c603 2014-08-24 06:15:44,821 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/39b8af357f05404077613e6bb6e5c603/FAMILY] 2014-08-24 06:15:44,824 INFO [IPC Server handler 7 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741877_1053 127.0.0.1:45654 127.0.0.1:39258 127.0.0.1:60268 2014-08-24 06:15:44,824 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/39b8af357f05404077613e6bb6e5c603 2014-08-24 06:15:44,825 INFO [IPC Server handler 9 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741872_1048 127.0.0.1:45654 127.0.0.1:39258 127.0.0.1:42916 2014-08-24 06:15:44,825 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(114): Table 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState' archived! 2014-08-24 06:15:44,825 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(117): Removing 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState' descriptor. 2014-08-24 06:15:44,825 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(121): Removing 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState' from region states. 2014-08-24 06:15:44,826 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(125): Marking 'testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState' as deleted. 2014-08-24 06:15:44,828 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState/write-master:578260000000002 2014-08-24 06:15:44,894 INFO [Thread-771] client.HBaseAdmin(705): Deleted testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState 2014-08-24 06:15:44,907 INFO [main] hbase.ResourceChecker(171): after: master.TestZKLessAMOnCluster#testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState Thread=487 (was 486) - Thread LEAK? -, OpenFileDescriptor=702 (was 661) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=65536 (was 65536), SystemLoadAverage=39 (was 17) - SystemLoadAverage LEAK? -, ProcessCount=152 (was 152), AvailableMemoryMB=27865 (was 27869), ConnectionCount=9 (was 9) 2014-08-24 06:15:44,919 INFO [main] hbase.ResourceChecker(147): before: master.TestZKLessAMOnCluster#testOfflineRegion Thread=487, OpenFileDescriptor=702, MaxFileDescriptor=65536, SystemLoadAverage=39, ProcessCount=152, AvailableMemoryMB=27864, ConnectionCount=9 2014-08-24 06:15:44,921 INFO [FifoRpcScheduler.handler7-thread-1] master.HMaster(1747): Client=apurtell//10.229.0.82 create 'testOfflineRegion', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2014-08-24 06:15:44,925 DEBUG [FifoRpcScheduler.handler7-thread-1] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testOfflineRegion/write-master:578260000000000 2014-08-24 06:15:44,931 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.CreateTableHandler(148): Create table testOfflineRegion 2014-08-24 06:15:44,939 INFO [IPC Server handler 9 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741878_1054{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:44,939 INFO [IPC Server handler 2 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741878_1054{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:44,940 INFO [IPC Server handler 3 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741878_1054 size 299 2014-08-24 06:15:44,941 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] util.FSTableDescriptors(640): Wrote descriptor into: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/.tabledesc/.tableinfo.0000000001 2014-08-24 06:15:44,942 INFO [RegionOpenAndInitThread-testOfflineRegion-1] regionserver.HRegion(4284): creating HRegion testOfflineRegion HTD == 'testOfflineRegion', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testOfflineRegion 2014-08-24 06:15:44,942 INFO [RegionOpenAndInitThread-testOfflineRegion-3] regionserver.HRegion(4284): creating HRegion testOfflineRegion HTD == 'testOfflineRegion', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testOfflineRegion 2014-08-24 06:15:44,942 INFO [RegionOpenAndInitThread-testOfflineRegion-2] regionserver.HRegion(4284): creating HRegion testOfflineRegion HTD == 'testOfflineRegion', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testOfflineRegion 2014-08-24 06:15:44,943 INFO [RegionOpenAndInitThread-testOfflineRegion-5] regionserver.HRegion(4284): creating HRegion testOfflineRegion HTD == 'testOfflineRegion', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testOfflineRegion 2014-08-24 06:15:44,943 INFO [RegionOpenAndInitThread-testOfflineRegion-4] regionserver.HRegion(4284): creating HRegion testOfflineRegion HTD == 'testOfflineRegion', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testOfflineRegion 2014-08-24 06:15:44,951 INFO [IPC Server handler 2 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741879_1055{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:44,953 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741879_1055{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:44,953 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741879_1055{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:44,954 INFO [IPC Server handler 8 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741880_1056{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW]]} size 0 2014-08-24 06:15:44,954 DEBUG [RegionOpenAndInitThread-testOfflineRegion-3] regionserver.HRegion(639): Instantiated testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. 2014-08-24 06:15:44,954 DEBUG [RegionOpenAndInitThread-testOfflineRegion-3] regionserver.HRegion(1109): Closing testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5.: disabling compactions & flushes 2014-08-24 06:15:44,954 INFO [IPC Server handler 1 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741880_1056 size 51 2014-08-24 06:15:44,955 DEBUG [RegionOpenAndInitThread-testOfflineRegion-3] regionserver.HRegion(1136): Updates disabled for region testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. 2014-08-24 06:15:44,955 INFO [IPC Server handler 9 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741880_1056 size 51 2014-08-24 06:15:44,955 INFO [RegionOpenAndInitThread-testOfflineRegion-3] regionserver.HRegion(1218): Closed testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. 2014-08-24 06:15:44,958 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741882_1058{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:44,958 INFO [IPC Server handler 1 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741882_1058{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:44,959 INFO [IPC Server handler 9 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741882_1058{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:44,959 DEBUG [RegionOpenAndInitThread-testOfflineRegion-1] regionserver.HRegion(639): Instantiated testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. 2014-08-24 06:15:44,959 DEBUG [RegionOpenAndInitThread-testOfflineRegion-1] regionserver.HRegion(1109): Closing testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da.: disabling compactions & flushes 2014-08-24 06:15:44,959 DEBUG [RegionOpenAndInitThread-testOfflineRegion-1] regionserver.HRegion(1136): Updates disabled for region testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. 2014-08-24 06:15:44,959 INFO [RegionOpenAndInitThread-testOfflineRegion-1] regionserver.HRegion(1218): Closed testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. 2014-08-24 06:15:44,959 INFO [IPC Server handler 3 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741881_1057{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:44,959 DEBUG [RegionOpenAndInitThread-testOfflineRegion-5] regionserver.HRegion(639): Instantiated testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. 2014-08-24 06:15:44,960 INFO [IPC Server handler 2 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741881_1057{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:44,960 DEBUG [RegionOpenAndInitThread-testOfflineRegion-5] regionserver.HRegion(1109): Closing testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f.: disabling compactions & flushes 2014-08-24 06:15:44,960 DEBUG [RegionOpenAndInitThread-testOfflineRegion-5] regionserver.HRegion(1136): Updates disabled for region testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. 2014-08-24 06:15:44,960 INFO [RegionOpenAndInitThread-testOfflineRegion-5] regionserver.HRegion(1218): Closed testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. 2014-08-24 06:15:44,960 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741883_1059{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:44,960 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741881_1057 size 52 2014-08-24 06:15:44,960 DEBUG [RegionOpenAndInitThread-testOfflineRegion-2] regionserver.HRegion(639): Instantiated testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. 2014-08-24 06:15:44,960 INFO [IPC Server handler 5 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741883_1059{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:44,960 DEBUG [RegionOpenAndInitThread-testOfflineRegion-2] regionserver.HRegion(1109): Closing testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2.: disabling compactions & flushes 2014-08-24 06:15:44,961 DEBUG [RegionOpenAndInitThread-testOfflineRegion-2] regionserver.HRegion(1136): Updates disabled for region testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. 2014-08-24 06:15:44,961 INFO [RegionOpenAndInitThread-testOfflineRegion-2] regionserver.HRegion(1218): Closed testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. 2014-08-24 06:15:44,961 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741883_1059 size 52 2014-08-24 06:15:44,999 DEBUG [RegionOpenAndInitThread-testOfflineRegion-4] regionserver.HRegion(639): Instantiated testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. 2014-08-24 06:15:44,999 DEBUG [RegionOpenAndInitThread-testOfflineRegion-4] regionserver.HRegion(1109): Closing testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a.: disabling compactions & flushes 2014-08-24 06:15:44,999 DEBUG [RegionOpenAndInitThread-testOfflineRegion-4] regionserver.HRegion(1136): Updates disabled for region testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. 2014-08-24 06:15:44,999 INFO [RegionOpenAndInitThread-testOfflineRegion-4] regionserver.HRegion(1218): Closed testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. 2014-08-24 06:15:45,004 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] catalog.MetaEditor(278): Added 5 2014-08-24 06:15:45,004 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.AssignmentManager(2696): Bulk assigning 5 region(s) across 3 server(s), round-robin=true 2014-08-24 06:15:45,005 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.AssignmentManager(1597): Assigning 2 region(s) to ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,005 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.AssignmentManager(1597): Assigning 2 region(s) to ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,005 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.GeneralBulkAssigner(177): Timeout-on-RIT=152000 2014-08-24 06:15:45,005 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.RegionStates(802): Transition {5fa8d7f89143dac7fd49a75d32df5fe5 state=OFFLINE, ts=1408860945004, server=null} to {5fa8d7f89143dac7fd49a75d32df5fe5 state=PENDING_OPEN, ts=1408860945005, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:45,005 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStates(802): Transition {dae32064329fb150ffdeb6870a3f39da state=OFFLINE, ts=1408860945004, server=null} to {dae32064329fb150ffdeb6870a3f39da state=PENDING_OPEN, ts=1408860945005, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:45,005 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.RegionStateStore(183): Updating row testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,005 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.AssignmentManager(1597): Assigning 1 region(s) to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,005 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStateStore(183): Updating row testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,005 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.RegionStates(802): Transition {82f7125fa7f90d16066a2aecf1d11c4f state=OFFLINE, ts=1408860945004, server=null} to {82f7125fa7f90d16066a2aecf1d11c4f state=PENDING_OPEN, ts=1408860945005, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,006 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.RegionStateStore(183): Updating row testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,007 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.RegionStates(802): Transition {b28a126e922415eb4df32098785d65c2 state=OFFLINE, ts=1408860945004, server=null} to {b28a126e922415eb4df32098785d65c2 state=PENDING_OPEN, ts=1408860945007, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:45,007 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.RegionStateStore(183): Updating row testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,010 INFO [PriorityRpcServer.handler=2,queue=0,port=58042] regionserver.HRegionServer(3718): Open testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. 2014-08-24 06:15:45,011 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStates(802): Transition {3929fd8be455c03fa0592b367cf2039a state=OFFLINE, ts=1408860945004, server=null} to {3929fd8be455c03fa0592b367cf2039a state=PENDING_OPEN, ts=1408860945011, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:45,011 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStateStore(183): Updating row testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,013 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.AssignmentManager(1777): Bulk assigning done for ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,013 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(4470): Opening region: {ENCODED => 82f7125fa7f90d16066a2aecf1d11c4f, NAME => 'testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f.', STARTKEY => 'Z', ENDKEY => ''} 2014-08-24 06:15:45,013 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-2] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:45,013 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testOfflineRegion 82f7125fa7f90d16066a2aecf1d11c4f 2014-08-24 06:15:45,013 INFO [PriorityRpcServer.handler=1,queue=0,port=55743] regionserver.HRegionServer(3718): Open testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. 2014-08-24 06:15:45,014 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(639): Instantiated testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. 2014-08-24 06:15:45,015 INFO [PriorityRpcServer.handler=7,queue=0,port=51857] regionserver.HRegionServer(3718): Open testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. 2014-08-24 06:15:45,017 INFO [PriorityRpcServer.handler=1,queue=0,port=55743] regionserver.HRegionServer(3718): Open testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. 2014-08-24 06:15:45,017 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(4470): Opening region: {ENCODED => 5fa8d7f89143dac7fd49a75d32df5fe5, NAME => 'testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5.', STARTKEY => 'I', ENDKEY => 'Q'} 2014-08-24 06:15:45,017 INFO [StoreOpener-82f7125fa7f90d16066a2aecf1d11c4f-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:45,017 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.AssignmentManager(1777): Bulk assigning done for ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,017 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(4470): Opening region: {ENCODED => b28a126e922415eb4df32098785d65c2, NAME => 'testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2.', STARTKEY => 'A', ENDKEY => 'I'} 2014-08-24 06:15:45,017 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-0] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:45,018 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-2] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:45,018 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testOfflineRegion 5fa8d7f89143dac7fd49a75d32df5fe5 2014-08-24 06:15:45,018 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testOfflineRegion b28a126e922415eb4df32098785d65c2 2014-08-24 06:15:45,018 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(639): Instantiated testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. 2014-08-24 06:15:45,018 DEBUG [StoreOpener-82f7125fa7f90d16066a2aecf1d11c4f-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testOfflineRegion/82f7125fa7f90d16066a2aecf1d11c4f/FAMILY 2014-08-24 06:15:45,018 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(639): Instantiated testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. 2014-08-24 06:15:45,019 INFO [PriorityRpcServer.handler=7,queue=0,port=51857] regionserver.HRegionServer(3718): Open testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. 2014-08-24 06:15:45,019 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(4470): Opening region: {ENCODED => dae32064329fb150ffdeb6870a3f39da, NAME => 'testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da.', STARTKEY => '', ENDKEY => 'A'} 2014-08-24 06:15:45,019 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testOfflineRegion/82f7125fa7f90d16066a2aecf1d11c4f 2014-08-24 06:15:45,019 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-2] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:45,019 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.AssignmentManager(1777): Bulk assigning done for ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,019 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testOfflineRegion dae32064329fb150ffdeb6870a3f39da 2014-08-24 06:15:45,019 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.GeneralBulkAssigner(153): bulk assigning total 5 regions to 3 servers, took 14ms, with 5 regions still in transition 2014-08-24 06:15:45,019 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(4470): Opening region: {ENCODED => 3929fd8be455c03fa0592b367cf2039a, NAME => 'testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a.', STARTKEY => 'Q', ENDKEY => 'Z'} 2014-08-24 06:15:45,020 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.AssignmentManager(2703): Bulk assigning done 2014-08-24 06:15:45,019 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(639): Instantiated testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. 2014-08-24 06:15:45,020 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-1] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:45,020 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testOfflineRegion 3929fd8be455c03fa0592b367cf2039a 2014-08-24 06:15:45,020 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(639): Instantiated testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. 2014-08-24 06:15:45,020 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(740): Onlined 82f7125fa7f90d16066a2aecf1d11c4f; next sequenceid=1 2014-08-24 06:15:45,021 INFO [PostOpenDeployTasks:82f7125fa7f90d16066a2aecf1d11c4f] regionserver.HRegionServer(1796): Post open deploy tasks for region=testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. 2014-08-24 06:15:45,021 DEBUG [FifoRpcScheduler.handler7-thread-4] master.AssignmentManager(4068): Got transition OPENED for {82f7125fa7f90d16066a2aecf1d11c4f state=PENDING_OPEN, ts=1408860945005, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,021 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(802): Transition {82f7125fa7f90d16066a2aecf1d11c4f state=PENDING_OPEN, ts=1408860945005, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {82f7125fa7f90d16066a2aecf1d11c4f state=OPEN, ts=1408860945021, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,021 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStateStore(183): Updating row testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,022 INFO [StoreOpener-5fa8d7f89143dac7fd49a75d32df5fe5-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:45,023 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testOfflineRegion/write-master:578260000000000 2014-08-24 06:15:45,023 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.CreateTableHandler(173): failed. null 2014-08-24 06:15:45,023 INFO [StoreOpener-b28a126e922415eb4df32098785d65c2-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:45,023 DEBUG [StoreOpener-5fa8d7f89143dac7fd49a75d32df5fe5-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testOfflineRegion/5fa8d7f89143dac7fd49a75d32df5fe5/FAMILY 2014-08-24 06:15:45,024 INFO [StoreOpener-dae32064329fb150ffdeb6870a3f39da-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:45,024 DEBUG [StoreOpener-b28a126e922415eb4df32098785d65c2-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testOfflineRegion/b28a126e922415eb4df32098785d65c2/FAMILY 2014-08-24 06:15:45,024 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testOfflineRegion/5fa8d7f89143dac7fd49a75d32df5fe5 2014-08-24 06:15:45,024 INFO [StoreOpener-3929fd8be455c03fa0592b367cf2039a-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:45,025 DEBUG [StoreOpener-dae32064329fb150ffdeb6870a3f39da-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testOfflineRegion/dae32064329fb150ffdeb6870a3f39da/FAMILY 2014-08-24 06:15:45,025 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testOfflineRegion/b28a126e922415eb4df32098785d65c2 2014-08-24 06:15:45,025 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(359): Onlined 82f7125fa7f90d16066a2aecf1d11c4f on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,026 DEBUG [StoreOpener-3929fd8be455c03fa0592b367cf2039a-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testOfflineRegion/3929fd8be455c03fa0592b367cf2039a/FAMILY 2014-08-24 06:15:45,026 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testOfflineRegion/dae32064329fb150ffdeb6870a3f39da 2014-08-24 06:15:45,026 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(740): Onlined 5fa8d7f89143dac7fd49a75d32df5fe5; next sequenceid=1 2014-08-24 06:15:45,026 INFO [PostOpenDeployTasks:5fa8d7f89143dac7fd49a75d32df5fe5] regionserver.HRegionServer(1796): Post open deploy tasks for region=testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. 2014-08-24 06:15:45,026 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testOfflineRegion/3929fd8be455c03fa0592b367cf2039a 2014-08-24 06:15:45,027 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(740): Onlined b28a126e922415eb4df32098785d65c2; next sequenceid=1 2014-08-24 06:15:45,027 DEBUG [FifoRpcScheduler.handler7-thread-3] master.AssignmentManager(4068): Got transition OPENED for {5fa8d7f89143dac7fd49a75d32df5fe5 state=PENDING_OPEN, ts=1408860945005, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,027 INFO [PostOpenDeployTasks:b28a126e922415eb4df32098785d65c2] regionserver.HRegionServer(1796): Post open deploy tasks for region=testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. 2014-08-24 06:15:45,027 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(802): Transition {5fa8d7f89143dac7fd49a75d32df5fe5 state=PENDING_OPEN, ts=1408860945005, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {5fa8d7f89143dac7fd49a75d32df5fe5 state=OPEN, ts=1408860945027, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:45,027 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStateStore(183): Updating row testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,027 DEBUG [FifoRpcScheduler.handler7-thread-5] master.AssignmentManager(4068): Got transition OPENED for {b28a126e922415eb4df32098785d65c2 state=PENDING_OPEN, ts=1408860945007, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,028 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(740): Onlined dae32064329fb150ffdeb6870a3f39da; next sequenceid=1 2014-08-24 06:15:45,028 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(802): Transition {b28a126e922415eb4df32098785d65c2 state=PENDING_OPEN, ts=1408860945007, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {b28a126e922415eb4df32098785d65c2 state=OPEN, ts=1408860945028, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:45,028 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStateStore(183): Updating row testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,028 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(740): Onlined 3929fd8be455c03fa0592b367cf2039a; next sequenceid=1 2014-08-24 06:15:45,028 INFO [PostOpenDeployTasks:dae32064329fb150ffdeb6870a3f39da] regionserver.HRegionServer(1796): Post open deploy tasks for region=testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. 2014-08-24 06:15:45,028 INFO [PostOpenDeployTasks:3929fd8be455c03fa0592b367cf2039a] regionserver.HRegionServer(1796): Post open deploy tasks for region=testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. 2014-08-24 06:15:45,028 INFO [PostOpenDeployTasks:82f7125fa7f90d16066a2aecf1d11c4f] regionserver.HRegionServer(1827): Finished post open deploy task for testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. 2014-08-24 06:15:45,028 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-2] handler.OpenRegionHandler(185): Opened testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,029 DEBUG [FifoRpcScheduler.handler7-thread-2] master.AssignmentManager(4068): Got transition OPENED for {dae32064329fb150ffdeb6870a3f39da state=PENDING_OPEN, ts=1408860945005, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,029 DEBUG [FifoRpcScheduler.handler7-thread-1] master.AssignmentManager(4068): Got transition OPENED for {3929fd8be455c03fa0592b367cf2039a state=PENDING_OPEN, ts=1408860945011, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,029 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(802): Transition {dae32064329fb150ffdeb6870a3f39da state=PENDING_OPEN, ts=1408860945005, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {dae32064329fb150ffdeb6870a3f39da state=OPEN, ts=1408860945029, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:45,029 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(802): Transition {3929fd8be455c03fa0592b367cf2039a state=PENDING_OPEN, ts=1408860945011, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {3929fd8be455c03fa0592b367cf2039a state=OPEN, ts=1408860945029, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:45,029 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStateStore(183): Updating row testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,029 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStateStore(183): Updating row testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,030 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(359): Onlined 5fa8d7f89143dac7fd49a75d32df5fe5 on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,032 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(359): Onlined 3929fd8be455c03fa0592b367cf2039a on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,032 INFO [PostOpenDeployTasks:5fa8d7f89143dac7fd49a75d32df5fe5] regionserver.HRegionServer(1827): Finished post open deploy task for testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. 2014-08-24 06:15:45,032 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-0] handler.OpenRegionHandler(185): Opened testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,033 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(359): Onlined dae32064329fb150ffdeb6870a3f39da on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,034 INFO [PostOpenDeployTasks:3929fd8be455c03fa0592b367cf2039a] regionserver.HRegionServer(1827): Finished post open deploy task for testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. 2014-08-24 06:15:45,034 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] handler.OpenRegionHandler(185): Opened testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,035 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(359): Onlined b28a126e922415eb4df32098785d65c2 on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,036 INFO [PostOpenDeployTasks:dae32064329fb150ffdeb6870a3f39da] regionserver.HRegionServer(1827): Finished post open deploy task for testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. 2014-08-24 06:15:45,036 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] handler.OpenRegionHandler(185): Opened testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,038 INFO [PostOpenDeployTasks:b28a126e922415eb4df32098785d65c2] regionserver.HRegionServer(1827): Finished post open deploy task for testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. 2014-08-24 06:15:45,038 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-2] handler.OpenRegionHandler(185): Opened testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,041 INFO [Thread-821] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:45,041 DEBUG [Thread-821] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@2f691df3 2014-08-24 06:15:45,042 DEBUG [Thread-821-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:45,043 DEBUG [Thread-821] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:45,044 DEBUG [Thread-821-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d0039 connected 2014-08-24 06:15:45,049 DEBUG [Thread-821] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@2f691df3 2014-08-24 06:15:45,056 INFO [FifoRpcScheduler.handler7-thread-3] master.HMaster(2683): Client=apurtell//10.229.0.82 offline testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. 2014-08-24 06:15:45,056 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(802): Transition {5fa8d7f89143dac7fd49a75d32df5fe5 state=OPEN, ts=1408860945027, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {5fa8d7f89143dac7fd49a75d32df5fe5 state=OFFLINE, ts=1408860945056, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:45,056 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStateStore(183): Updating row testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. with state=OFFLINE 2014-08-24 06:15:45,058 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(455): Offlined 5fa8d7f89143dac7fd49a75d32df5fe5 from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,059 INFO [Thread-821] client.HBaseAdmin$5(917): Started disable of testOfflineRegion 2014-08-24 06:15:45,060 INFO [FifoRpcScheduler.handler7-thread-2] master.HMaster(1979): Client=apurtell//10.229.0.82 disable testOfflineRegion 2014-08-24 06:15:45,062 DEBUG [FifoRpcScheduler.handler7-thread-2] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testOfflineRegion/write-master:578260000000001 2014-08-24 06:15:45,067 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(130): Attempting to disable table testOfflineRegion 2014-08-24 06:15:45,068 INFO [Thread-821] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:45,068 DEBUG [Thread-821] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@6bdf5f43 2014-08-24 06:15:45,069 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(174): Offlining 4 regions. 2014-08-24 06:15:45,069 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2420): Starting unassign of testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. (offlining), current state: {dae32064329fb150ffdeb6870a3f39da state=OPEN, ts=1408860945029, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:45,069 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(802): Transition {dae32064329fb150ffdeb6870a3f39da state=OPEN, ts=1408860945029, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {dae32064329fb150ffdeb6870a3f39da state=PENDING_CLOSE, ts=1408860945069, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:45,069 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.AssignmentManager(2420): Starting unassign of testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. (offlining), current state: {b28a126e922415eb4df32098785d65c2 state=OPEN, ts=1408860945028, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:45,069 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-2] master.AssignmentManager(2420): Starting unassign of testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. (offlining), current state: {3929fd8be455c03fa0592b367cf2039a state=OPEN, ts=1408860945029, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:45,069 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStateStore(183): Updating row testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. with state=PENDING_CLOSE 2014-08-24 06:15:45,070 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-2] master.RegionStates(802): Transition {3929fd8be455c03fa0592b367cf2039a state=OPEN, ts=1408860945029, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {3929fd8be455c03fa0592b367cf2039a state=PENDING_CLOSE, ts=1408860945070, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:45,069 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-3] master.AssignmentManager(2420): Starting unassign of testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. (offlining), current state: {82f7125fa7f90d16066a2aecf1d11c4f state=OPEN, ts=1408860945021, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,069 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.RegionStates(802): Transition {b28a126e922415eb4df32098785d65c2 state=OPEN, ts=1408860945028, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {b28a126e922415eb4df32098785d65c2 state=PENDING_CLOSE, ts=1408860945069, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:45,070 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-3] master.RegionStates(802): Transition {82f7125fa7f90d16066a2aecf1d11c4f state=OPEN, ts=1408860945021, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {82f7125fa7f90d16066a2aecf1d11c4f state=PENDING_CLOSE, ts=1408860945070, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,070 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-2] master.RegionStateStore(183): Updating row testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. with state=PENDING_CLOSE 2014-08-24 06:15:45,070 DEBUG [Thread-821-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:45,070 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-3] master.RegionStateStore(183): Updating row testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. with state=PENDING_CLOSE 2014-08-24 06:15:45,070 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.RegionStateStore(183): Updating row testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. with state=PENDING_CLOSE 2014-08-24 06:15:45,070 DEBUG [Thread-821] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:45,071 DEBUG [Thread-821-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d003a connected 2014-08-24 06:15:45,072 INFO [PriorityRpcServer.handler=5,queue=0,port=51857] regionserver.HRegionServer(3859): Close dae32064329fb150ffdeb6870a3f39da, via zk=no, znode version=-1, on null 2014-08-24 06:15:45,072 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] handler.CloseRegionHandler(128): Processing close of testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. 2014-08-24 06:15:45,072 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 for region testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. 2014-08-24 06:15:45,073 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(1109): Closing testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da.: disabling compactions & flushes 2014-08-24 06:15:45,073 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(1136): Updates disabled for region testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. 2014-08-24 06:15:45,074 INFO [StoreCloserThread-testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:45,074 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:45,074 INFO [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(1218): Closed testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. 2014-08-24 06:15:45,074 DEBUG [FifoRpcScheduler.handler7-thread-5] master.AssignmentManager(4068): Got transition CLOSED for {dae32064329fb150ffdeb6870a3f39da state=PENDING_CLOSE, ts=1408860945069, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,075 INFO [PriorityRpcServer.handler=3,queue=0,port=55743] regionserver.HRegionServer(3859): Close b28a126e922415eb4df32098785d65c2, via zk=no, znode version=-1, on null 2014-08-24 06:15:45,075 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] handler.CloseRegionHandler(128): Processing close of testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. 2014-08-24 06:15:45,075 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 for region testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. 2014-08-24 06:15:45,075 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(1109): Closing testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2.: disabling compactions & flushes 2014-08-24 06:15:45,076 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(1136): Updates disabled for region testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. 2014-08-24 06:15:45,076 INFO [StoreCloserThread-testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:45,076 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:45,076 INFO [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(1218): Closed testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. 2014-08-24 06:15:45,077 DEBUG [Thread-821] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@6bdf5f43 2014-08-24 06:15:45,077 DEBUG [FifoRpcScheduler.handler7-thread-4] master.AssignmentManager(4068): Got transition CLOSED for {b28a126e922415eb4df32098785d65c2 state=PENDING_CLOSE, ts=1408860945069, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,077 INFO [PriorityRpcServer.handler=3,queue=0,port=58042] regionserver.HRegionServer(3859): Close 82f7125fa7f90d16066a2aecf1d11c4f, via zk=no, znode version=-1, on null 2014-08-24 06:15:45,077 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] handler.CloseRegionHandler(128): Processing close of testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. 2014-08-24 06:15:45,077 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-3] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 for region testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. 2014-08-24 06:15:45,078 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(1109): Closing testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f.: disabling compactions & flushes 2014-08-24 06:15:45,078 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(1136): Updates disabled for region testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. 2014-08-24 06:15:45,078 DEBUG [Thread-821] client.HBaseAdmin(956): Sleeping= 100ms, waiting for all regions to be disabled in testOfflineRegion 2014-08-24 06:15:45,078 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(802): Transition {dae32064329fb150ffdeb6870a3f39da state=PENDING_CLOSE, ts=1408860945069, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {dae32064329fb150ffdeb6870a3f39da state=OFFLINE, ts=1408860945078, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:45,078 INFO [StoreCloserThread-testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:45,078 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStateStore(183): Updating row testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. with state=OFFLINE 2014-08-24 06:15:45,079 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:45,079 INFO [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(1218): Closed testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. 2014-08-24 06:15:45,079 DEBUG [FifoRpcScheduler.handler7-thread-3] master.AssignmentManager(4068): Got transition CLOSED for {82f7125fa7f90d16066a2aecf1d11c4f state=PENDING_CLOSE, ts=1408860945070, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,080 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(802): Transition {b28a126e922415eb4df32098785d65c2 state=PENDING_CLOSE, ts=1408860945069, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {b28a126e922415eb4df32098785d65c2 state=OFFLINE, ts=1408860945080, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:45,080 INFO [PriorityRpcServer.handler=4,queue=0,port=51857] regionserver.HRegionServer(3859): Close 3929fd8be455c03fa0592b367cf2039a, via zk=no, znode version=-1, on null 2014-08-24 06:15:45,080 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStateStore(183): Updating row testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. with state=OFFLINE 2014-08-24 06:15:45,080 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] handler.CloseRegionHandler(128): Processing close of testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. 2014-08-24 06:15:45,080 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-2] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 for region testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. 2014-08-24 06:15:45,081 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(1109): Closing testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a.: disabling compactions & flushes 2014-08-24 06:15:45,081 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(1136): Updates disabled for region testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. 2014-08-24 06:15:45,081 INFO [StoreCloserThread-testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:45,081 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:45,081 INFO [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(1218): Closed testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. 2014-08-24 06:15:45,082 DEBUG [FifoRpcScheduler.handler7-thread-1] master.AssignmentManager(4068): Got transition CLOSED for {3929fd8be455c03fa0592b367cf2039a state=PENDING_CLOSE, ts=1408860945070, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,082 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(802): Transition {82f7125fa7f90d16066a2aecf1d11c4f state=PENDING_CLOSE, ts=1408860945070, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {82f7125fa7f90d16066a2aecf1d11c4f state=OFFLINE, ts=1408860945082, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,082 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStateStore(183): Updating row testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. with state=OFFLINE 2014-08-24 06:15:45,082 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(455): Offlined dae32064329fb150ffdeb6870a3f39da from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,082 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] handler.CloseRegionHandler(182): Closed testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. 2014-08-24 06:15:45,084 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(455): Offlined b28a126e922415eb4df32098785d65c2 from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,084 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(802): Transition {3929fd8be455c03fa0592b367cf2039a state=PENDING_CLOSE, ts=1408860945070, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {3929fd8be455c03fa0592b367cf2039a state=OFFLINE, ts=1408860945084, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:45,084 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] handler.CloseRegionHandler(182): Closed testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. 2014-08-24 06:15:45,084 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStateStore(183): Updating row testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. with state=OFFLINE 2014-08-24 06:15:45,085 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(455): Offlined 82f7125fa7f90d16066a2aecf1d11c4f from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,086 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] handler.CloseRegionHandler(182): Closed testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. 2014-08-24 06:15:45,087 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(455): Offlined 3929fd8be455c03fa0592b367cf2039a from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,087 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] handler.CloseRegionHandler(182): Closed testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. 2014-08-24 06:15:45,120 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler$BulkDisabler(230): Disable waiting until done; 300000 ms remaining; [] 2014-08-24 06:15:45,121 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(190): Disabled table, testOfflineRegion, is done=true 2014-08-24 06:15:45,122 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testOfflineRegion/write-master:578260000000001 2014-08-24 06:15:45,179 INFO [Thread-821] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:45,179 DEBUG [Thread-821] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@e93e12d 2014-08-24 06:15:45,180 DEBUG [Thread-821-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:45,181 DEBUG [Thread-821] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:45,182 DEBUG [Thread-821-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d003b connected 2014-08-24 06:15:45,185 DEBUG [Thread-821] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@e93e12d 2014-08-24 06:15:45,187 INFO [Thread-821] client.HBaseAdmin(971): Disabled testOfflineRegion 2014-08-24 06:15:45,187 INFO [FifoRpcScheduler.handler7-thread-5] master.HMaster(1816): Client=apurtell//10.229.0.82 delete testOfflineRegion 2014-08-24 06:15:45,189 DEBUG [FifoRpcScheduler.handler7-thread-5] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testOfflineRegion/write-master:578260000000002 2014-08-24 06:15:45,196 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.TableEventHandler(123): Handling table operation C_M_DELETE_TABLE on table testOfflineRegion 2014-08-24 06:15:45,201 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(93): Deleting regions from META 2014-08-24 06:15:45,204 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] catalog.MetaEditor(495): Deleted [{ENCODED => dae32064329fb150ffdeb6870a3f39da, NAME => 'testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da.', STARTKEY => '', ENDKEY => 'A'}, {ENCODED => b28a126e922415eb4df32098785d65c2, NAME => 'testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2.', STARTKEY => 'A', ENDKEY => 'I'}, {ENCODED => 5fa8d7f89143dac7fd49a75d32df5fe5, NAME => 'testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5.', STARTKEY => 'I', ENDKEY => 'Q'}, {ENCODED => 3929fd8be455c03fa0592b367cf2039a, NAME => 'testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a.', STARTKEY => 'Q', ENDKEY => 'Z'}, {ENCODED => 82f7125fa7f90d16066a2aecf1d11c4f, NAME => 'testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f.', STARTKEY => 'Z', ENDKEY => ''}] 2014-08-24 06:15:45,206 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testOfflineRegion,,1408860944920.dae32064329fb150ffdeb6870a3f39da. from FS 2014-08-24 06:15:45,206 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/dae32064329fb150ffdeb6870a3f39da 2014-08-24 06:15:45,207 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/dae32064329fb150ffdeb6870a3f39da/FAMILY] 2014-08-24 06:15:45,210 INFO [IPC Server handler 1 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741880_1056 127.0.0.1:60268 127.0.0.1:45654 127.0.0.1:39258 2014-08-24 06:15:45,211 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/dae32064329fb150ffdeb6870a3f39da 2014-08-24 06:15:45,212 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testOfflineRegion,A,1408860944921.b28a126e922415eb4df32098785d65c2. from FS 2014-08-24 06:15:45,212 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/b28a126e922415eb4df32098785d65c2 2014-08-24 06:15:45,213 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/b28a126e922415eb4df32098785d65c2/FAMILY] 2014-08-24 06:15:45,216 INFO [IPC Server handler 8 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741881_1057 127.0.0.1:60268 127.0.0.1:45654 127.0.0.1:39258 2014-08-24 06:15:45,216 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/b28a126e922415eb4df32098785d65c2 2014-08-24 06:15:45,216 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. from FS 2014-08-24 06:15:45,216 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/5fa8d7f89143dac7fd49a75d32df5fe5 2014-08-24 06:15:45,218 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/5fa8d7f89143dac7fd49a75d32df5fe5/FAMILY] 2014-08-24 06:15:45,221 INFO [IPC Server handler 5 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741879_1055 127.0.0.1:39258 127.0.0.1:45654 127.0.0.1:42916 2014-08-24 06:15:45,221 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/5fa8d7f89143dac7fd49a75d32df5fe5 2014-08-24 06:15:45,221 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testOfflineRegion,Q,1408860944921.3929fd8be455c03fa0592b367cf2039a. from FS 2014-08-24 06:15:45,221 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/3929fd8be455c03fa0592b367cf2039a 2014-08-24 06:15:45,223 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/3929fd8be455c03fa0592b367cf2039a/FAMILY] 2014-08-24 06:15:45,226 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741883_1059 127.0.0.1:42916 127.0.0.1:39258 127.0.0.1:45654 2014-08-24 06:15:45,226 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/3929fd8be455c03fa0592b367cf2039a 2014-08-24 06:15:45,226 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testOfflineRegion,Z,1408860944921.82f7125fa7f90d16066a2aecf1d11c4f. from FS 2014-08-24 06:15:45,226 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/82f7125fa7f90d16066a2aecf1d11c4f 2014-08-24 06:15:45,228 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/82f7125fa7f90d16066a2aecf1d11c4f/FAMILY] 2014-08-24 06:15:45,231 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741882_1058 127.0.0.1:42916 127.0.0.1:39258 127.0.0.1:60268 2014-08-24 06:15:45,231 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testOfflineRegion/82f7125fa7f90d16066a2aecf1d11c4f 2014-08-24 06:15:45,231 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741878_1054 127.0.0.1:42916 127.0.0.1:60268 127.0.0.1:39258 2014-08-24 06:15:45,232 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(114): Table 'testOfflineRegion' archived! 2014-08-24 06:15:45,232 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(117): Removing 'testOfflineRegion' descriptor. 2014-08-24 06:15:45,232 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(121): Removing 'testOfflineRegion' from region states. 2014-08-24 06:15:45,232 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(125): Marking 'testOfflineRegion' as deleted. 2014-08-24 06:15:45,234 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testOfflineRegion/write-master:578260000000002 2014-08-24 06:15:45,299 INFO [Thread-821] client.HBaseAdmin(705): Deleted testOfflineRegion 2014-08-24 06:15:45,312 INFO [main] hbase.ResourceChecker(171): after: master.TestZKLessAMOnCluster#testOfflineRegion Thread=486 (was 487), OpenFileDescriptor=702 (was 702), MaxFileDescriptor=65536 (was 65536), SystemLoadAverage=39 (was 39), ProcessCount=152 (was 152), AvailableMemoryMB=27863 (was 27864), ConnectionCount=9 (was 9) 2014-08-24 06:15:45,324 INFO [main] hbase.ResourceChecker(147): before: master.TestZKLessAMOnCluster#testForceAssignWhileClosing Thread=486, OpenFileDescriptor=702, MaxFileDescriptor=65536, SystemLoadAverage=39, ProcessCount=152, AvailableMemoryMB=27863, ConnectionCount=9 2014-08-24 06:15:45,326 INFO [FifoRpcScheduler.handler7-thread-2] master.HMaster(1747): Client=apurtell//10.229.0.82 create 'testForceAssignWhileClosing', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2014-08-24 06:15:45,330 DEBUG [FifoRpcScheduler.handler7-thread-2] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testForceAssignWhileClosing/write-master:578260000000000 2014-08-24 06:15:45,337 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.CreateTableHandler(148): Create table testForceAssignWhileClosing 2014-08-24 06:15:45,345 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741884_1060{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:45,346 INFO [IPC Server handler 5 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741884_1060{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:45,346 INFO [IPC Server handler 1 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741884_1060 size 309 2014-08-24 06:15:45,347 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] util.FSTableDescriptors(640): Wrote descriptor into: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testForceAssignWhileClosing/.tabledesc/.tableinfo.0000000001 2014-08-24 06:15:45,349 INFO [RegionOpenAndInitThread-testForceAssignWhileClosing-1] regionserver.HRegion(4284): creating HRegion testForceAssignWhileClosing HTD == 'testForceAssignWhileClosing', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testForceAssignWhileClosing 2014-08-24 06:15:45,355 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741885_1061{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW]]} size 0 2014-08-24 06:15:45,355 INFO [IPC Server handler 9 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741885_1061{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW]]} size 0 2014-08-24 06:15:45,356 INFO [IPC Server handler 7 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741885_1061 size 60 2014-08-24 06:15:45,356 DEBUG [RegionOpenAndInitThread-testForceAssignWhileClosing-1] regionserver.HRegion(639): Instantiated testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. 2014-08-24 06:15:45,357 DEBUG [RegionOpenAndInitThread-testForceAssignWhileClosing-1] regionserver.HRegion(1109): Closing testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b.: disabling compactions & flushes 2014-08-24 06:15:45,357 DEBUG [RegionOpenAndInitThread-testForceAssignWhileClosing-1] regionserver.HRegion(1136): Updates disabled for region testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. 2014-08-24 06:15:45,357 INFO [RegionOpenAndInitThread-testForceAssignWhileClosing-1] regionserver.HRegion(1218): Closed testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. 2014-08-24 06:15:45,360 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] catalog.MetaEditor(278): Added 1 2014-08-24 06:15:45,360 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.AssignmentManager(2696): Bulk assigning 1 region(s) across 3 server(s), round-robin=true 2014-08-24 06:15:45,360 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.AssignmentManager(1597): Assigning 1 region(s) to ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,360 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.GeneralBulkAssigner(177): Timeout-on-RIT=151000 2014-08-24 06:15:45,360 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.RegionStates(802): Transition {d6c1e0cbf48826990b4d1ae53a5dc74b state=OFFLINE, ts=1408860945360, server=null} to {d6c1e0cbf48826990b4d1ae53a5dc74b state=PENDING_OPEN, ts=1408860945360, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:45,361 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.RegionStateStore(183): Updating row testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,363 INFO [PriorityRpcServer.handler=4,queue=0,port=55743] regionserver.HRegionServer(3718): Open testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. 2014-08-24 06:15:45,366 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.AssignmentManager(1777): Bulk assigning done for ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,366 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(4470): Opening region: {ENCODED => d6c1e0cbf48826990b4d1ae53a5dc74b, NAME => 'testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b.', STARTKEY => '', ENDKEY => ''} 2014-08-24 06:15:45,366 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.GeneralBulkAssigner(153): bulk assigning total 1 regions to 3 servers, took 5ms, with 1 regions still in transition 2014-08-24 06:15:45,366 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.AssignmentManager(2703): Bulk assigning done 2014-08-24 06:15:45,366 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-1] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:45,366 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testForceAssignWhileClosing d6c1e0cbf48826990b4d1ae53a5dc74b 2014-08-24 06:15:45,366 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(639): Instantiated testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. 2014-08-24 06:15:45,369 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testForceAssignWhileClosing/write-master:578260000000000 2014-08-24 06:15:45,369 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.CreateTableHandler(173): failed. null 2014-08-24 06:15:45,369 INFO [StoreOpener-d6c1e0cbf48826990b4d1ae53a5dc74b-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:45,370 DEBUG [StoreOpener-d6c1e0cbf48826990b4d1ae53a5dc74b-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testForceAssignWhileClosing/d6c1e0cbf48826990b4d1ae53a5dc74b/FAMILY 2014-08-24 06:15:45,371 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testForceAssignWhileClosing/d6c1e0cbf48826990b4d1ae53a5dc74b 2014-08-24 06:15:45,372 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(740): Onlined d6c1e0cbf48826990b4d1ae53a5dc74b; next sequenceid=1 2014-08-24 06:15:45,372 INFO [PostOpenDeployTasks:d6c1e0cbf48826990b4d1ae53a5dc74b] regionserver.HRegionServer(1796): Post open deploy tasks for region=testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. 2014-08-24 06:15:45,373 DEBUG [FifoRpcScheduler.handler7-thread-1] master.AssignmentManager(4068): Got transition OPENED for {d6c1e0cbf48826990b4d1ae53a5dc74b state=PENDING_OPEN, ts=1408860945360, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,373 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(802): Transition {d6c1e0cbf48826990b4d1ae53a5dc74b state=PENDING_OPEN, ts=1408860945360, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {d6c1e0cbf48826990b4d1ae53a5dc74b state=OPEN, ts=1408860945373, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:45,373 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStateStore(183): Updating row testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,375 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(359): Onlined d6c1e0cbf48826990b4d1ae53a5dc74b on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,377 INFO [PostOpenDeployTasks:d6c1e0cbf48826990b4d1ae53a5dc74b] regionserver.HRegionServer(1827): Finished post open deploy task for testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. 2014-08-24 06:15:45,377 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-1] handler.OpenRegionHandler(185): Opened testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,448 INFO [Thread-871] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:45,449 DEBUG [Thread-871] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@7edce77f 2014-08-24 06:15:45,450 DEBUG [Thread-871-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:45,450 DEBUG [Thread-871] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:45,451 DEBUG [Thread-871-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d003c connected 2014-08-24 06:15:45,456 DEBUG [Thread-871] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@7edce77f 2014-08-24 06:15:45,459 DEBUG [Thread-871] catalog.MetaEditor(238): Added testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,459 WARN [Thread-871] master.AssignmentManager(1918): Assigning a region not in region states: {ENCODED => c5ff407162be4ce7c7d143f14616a9dc, NAME => 'testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc.', STARTKEY => 'A', ENDKEY => 'Z'} 2014-08-24 06:15:45,459 DEBUG [Thread-871] master.AssignmentManager(2372): No previous transition plan found (or ignoring an existing plan) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc.; generated random plan=hri=testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., src=, dest=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571; 3 (online=3, available=3) available servers, forceNewPlan=false 2014-08-24 06:15:45,459 INFO [Thread-871] master.AssignmentManager(2083): Assigning testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,460 INFO [Thread-871] master.RegionStates(802): Transition {c5ff407162be4ce7c7d143f14616a9dc state=OFFLINE, ts=1408860945459, server=null} to {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_OPEN, ts=1408860945460, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,460 INFO [Thread-871] master.RegionStateStore(183): Updating row testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,462 INFO [PriorityRpcServer.handler=4,queue=0,port=58042] regionserver.HRegionServer(3718): Open testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,465 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(4470): Opening region: {ENCODED => c5ff407162be4ce7c7d143f14616a9dc, NAME => 'testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc.', STARTKEY => 'A', ENDKEY => 'Z'} 2014-08-24 06:15:45,465 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-1] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:45,465 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testForceAssignWhileClosing c5ff407162be4ce7c7d143f14616a9dc 2014-08-24 06:15:45,465 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(639): Instantiated testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,466 WARN [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegionFileSystem(788): .regioninfo file not found for region: c5ff407162be4ce7c7d143f14616a9dc 2014-08-24 06:15:45,472 INFO [IPC Server handler 3 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741886_1062{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:45,473 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741886_1062{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:45,473 INFO [IPC Server handler 5 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741886_1062 size 62 2014-08-24 06:15:45,476 INFO [StoreOpener-c5ff407162be4ce7c7d143f14616a9dc-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:45,477 DEBUG [StoreOpener-c5ff407162be4ce7c7d143f14616a9dc-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testForceAssignWhileClosing/c5ff407162be4ce7c7d143f14616a9dc/FAMILY 2014-08-24 06:15:45,478 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testForceAssignWhileClosing/c5ff407162be4ce7c7d143f14616a9dc 2014-08-24 06:15:45,479 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(740): Onlined c5ff407162be4ce7c7d143f14616a9dc; next sequenceid=1 2014-08-24 06:15:45,479 INFO [PostOpenDeployTasks:c5ff407162be4ce7c7d143f14616a9dc] regionserver.HRegionServer(1796): Post open deploy tasks for region=testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,479 DEBUG [FifoRpcScheduler.handler7-thread-2] master.AssignmentManager(4068): Got transition OPENED for {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_OPEN, ts=1408860945460, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,480 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(802): Transition {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_OPEN, ts=1408860945460, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {c5ff407162be4ce7c7d143f14616a9dc state=OPEN, ts=1408860945480, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,480 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStateStore(183): Updating row testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,482 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(359): Onlined c5ff407162be4ce7c7d143f14616a9dc on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,482 DEBUG [Thread-871] master.AssignmentManager(2420): Starting unassign of testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. (offlining), current state: {c5ff407162be4ce7c7d143f14616a9dc state=OPEN, ts=1408860945480, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,482 INFO [Thread-871] master.RegionStates(802): Transition {c5ff407162be4ce7c7d143f14616a9dc state=OPEN, ts=1408860945480, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_CLOSE, ts=1408860945482, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,482 INFO [Thread-871] master.RegionStateStore(183): Updating row testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. with state=PENDING_CLOSE 2014-08-24 06:15:45,484 INFO [PostOpenDeployTasks:c5ff407162be4ce7c7d143f14616a9dc] regionserver.HRegionServer(1827): Finished post open deploy task for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,484 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] handler.OpenRegionHandler(185): Opened testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,485 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=1 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,485 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=2 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,486 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=3 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,487 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=4 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,487 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=5 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,488 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=6 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,489 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=7 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,489 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=8 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,490 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=9 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,491 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=10 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,492 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=11 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,492 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=12 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,493 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=13 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,493 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=14 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,494 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=15 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,495 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=16 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,495 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=17 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,496 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=18 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,497 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=19 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,497 INFO [Thread-871] master.AssignmentManager(1898): Server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 returned java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., try=20 of 20 java.io.IOException: java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.GeneratedConstructorAccessor34.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2513) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2525) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:2400) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:385) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: fail preClose from coprocessor at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver.preClose(TestAssignmentManagerOnCluster.java:945) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$4.call(RegionCoprocessorHost.java:364) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1517) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1592) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preClose(RegionCoprocessorHost.java:360) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3855) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 15 more 2014-08-24 06:15:45,498 WARN [Thread-871] master.RegionStates(793): Failed to open/close c5ff407162be4ce7c7d143f14616a9dc on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571, set to FAILED_CLOSE 2014-08-24 06:15:45,498 INFO [Thread-871] master.RegionStates(802): Transition {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_CLOSE, ts=1408860945482, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {c5ff407162be4ce7c7d143f14616a9dc state=FAILED_CLOSE, ts=1408860945498, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,498 INFO [Thread-871] master.RegionStateStore(183): Updating row testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. with state=FAILED_CLOSE 2014-08-24 06:15:45,500 DEBUG [Thread-871] master.AssignmentManager(2420): Starting unassign of testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. (offlining), current state: {c5ff407162be4ce7c7d143f14616a9dc state=FAILED_CLOSE, ts=1408860945498, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,500 DEBUG [Thread-871] master.AssignmentManager(2499): Attempting to unassign testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. which is already FAILED_CLOSE but forcing to send a CLOSE RPC again 2014-08-24 06:15:45,500 INFO [Thread-871] master.RegionStates(802): Transition {c5ff407162be4ce7c7d143f14616a9dc state=FAILED_CLOSE, ts=1408860945498, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_CLOSE, ts=1408860945500, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,500 INFO [Thread-871] master.RegionStateStore(183): Updating row testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. with state=PENDING_CLOSE 2014-08-24 06:15:45,502 INFO [PriorityRpcServer.handler=5,queue=0,port=58042] regionserver.HRegionServer(3859): Close c5ff407162be4ce7c7d143f14616a9dc, via zk=no, znode version=-1, on null 2014-08-24 06:15:45,503 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] handler.CloseRegionHandler(128): Processing close of testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,503 DEBUG [Thread-871] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 for region testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,504 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(1109): Closing testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc.: disabling compactions & flushes 2014-08-24 06:15:45,504 DEBUG [Thread-871] master.AssignmentManager(1924): Force region state offline {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_CLOSE, ts=1408860945502, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,504 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(1136): Updates disabled for region testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,504 INFO [PriorityRpcServer.handler=6,queue=0,port=58042] regionserver.HRegionServer(3859): Close c5ff407162be4ce7c7d143f14616a9dc, via zk=no, znode version=-1, on null 2014-08-24 06:15:45,504 INFO [PriorityRpcServer.handler=6,queue=0,port=58042] regionserver.HRegionServer(2655): Received CLOSE for the region: c5ff407162be4ce7c7d143f14616a9dc ,which we are already trying to CLOSE, but not completed yet 2014-08-24 06:15:45,504 INFO [StoreCloserThread-testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:45,504 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:45,505 INFO [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(1218): Closed testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,505 DEBUG [Thread-871] master.AssignmentManager(1865): update {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_CLOSE, ts=1408860945502, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} the timestamp. 2014-08-24 06:15:45,505 DEBUG [Thread-871] master.AssignmentManager(1875): Region is already in transition; waiting up to 1000ms org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region c5ff407162be4ce7c7d143f14616a9dc was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2663) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3862) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:395) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException): org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region c5ff407162be4ce7c7d143f14616a9dc was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2663) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3862) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 14 more 2014-08-24 06:15:45,505 DEBUG [FifoRpcScheduler.handler7-thread-5] master.AssignmentManager(4068): Got transition CLOSED for {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_CLOSE, ts=1408860945505, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,507 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(802): Transition {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_CLOSE, ts=1408860945505, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {c5ff407162be4ce7c7d143f14616a9dc state=CLOSED, ts=1408860945507, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,507 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStateStore(183): Updating row testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. with state=CLOSED 2014-08-24 06:15:45,509 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] handler.CloseRegionHandler(182): Closed testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,605 INFO [PriorityRpcServer.handler=7,queue=0,port=58042] regionserver.HRegionServer(3859): Close c5ff407162be4ce7c7d143f14616a9dc, via zk=no, znode version=-1, on null 2014-08-24 06:15:45,606 ERROR [PriorityRpcServer.handler=7,queue=0,port=58042] regionserver.HRegionServer(2668): Received CLOSE for a region which is not online, and we're not opening. 2014-08-24 06:15:45,606 DEBUG [Thread-871] master.AssignmentManager(1847): Offline testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., it's not any more on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: The region c5ff407162be4ce7c7d143f14616a9dc is not online, and is not opening. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2671) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3862) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:285) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1666) at org.apache.hadoop.hbase.master.ServerManager.sendRegionClose(ServerManager.java:731) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1823) at org.apache.hadoop.hbase.master.AssignmentManager.forceRegionStateToOffline(AssignmentManager.java:1940) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1569) at org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testForceAssignWhileClosing(TestAssignmentManagerOnCluster.java:395) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException): org.apache.hadoop.hbase.NotServingRegionException: The region c5ff407162be4ce7c7d143f14616a9dc is not online, and is not opening. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2671) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3862) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20166) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1457) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.closeRegion(AdminProtos.java:20976) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.closeRegion(ProtobufUtil.java:1663) ... 14 more 2014-08-24 06:15:45,606 INFO [Thread-871] master.RegionStates(802): Transition {c5ff407162be4ce7c7d143f14616a9dc state=CLOSED, ts=1408860945507, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {c5ff407162be4ce7c7d143f14616a9dc state=OFFLINE, ts=1408860945606, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,606 INFO [Thread-871] master.RegionStateStore(183): Updating row testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. with state=OFFLINE 2014-08-24 06:15:45,608 INFO [Thread-871] master.RegionStates(455): Offlined c5ff407162be4ce7c7d143f14616a9dc from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,608 DEBUG [Thread-871] master.AssignmentManager(2372): No previous transition plan found (or ignoring an existing plan) for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc.; generated random plan=hri=testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc., src=, dest=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571; 3 (online=3, available=3) available servers, forceNewPlan=true 2014-08-24 06:15:45,608 INFO [Thread-871] master.AssignmentManager(2083): Assigning testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,608 INFO [Thread-871] master.RegionStates(802): Transition {c5ff407162be4ce7c7d143f14616a9dc state=OFFLINE, ts=1408860945606, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_OPEN, ts=1408860945608, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,608 INFO [Thread-871] master.RegionStateStore(183): Updating row testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. with state=PENDING_OPEN 2014-08-24 06:15:45,611 INFO [PriorityRpcServer.handler=8,queue=0,port=58042] regionserver.HRegionServer(3718): Open testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,612 INFO [Thread-871] master.AssignmentManager(3131): Waiting for c5ff407162be4ce7c7d143f14616a9dc to leave regions-in-transition, timeOut=-1 ms. 2014-08-24 06:15:45,612 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(4470): Opening region: {ENCODED => c5ff407162be4ce7c7d143f14616a9dc, NAME => 'testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc.', STARTKEY => 'A', ENDKEY => 'Z'} 2014-08-24 06:15:45,612 DEBUG [AM.-pool37-t23] master.AssignmentManager(1934): Skip assigning {ENCODED => c5ff407162be4ce7c7d143f14616a9dc, NAME => 'testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc.', STARTKEY => 'A', ENDKEY => 'Z'}, it is already {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_OPEN, ts=1408860945608, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,612 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-0] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:45,612 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testForceAssignWhileClosing c5ff407162be4ce7c7d143f14616a9dc 2014-08-24 06:15:45,613 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(639): Instantiated testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,615 INFO [StoreOpener-c5ff407162be4ce7c7d143f14616a9dc-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:45,616 DEBUG [StoreOpener-c5ff407162be4ce7c7d143f14616a9dc-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testForceAssignWhileClosing/c5ff407162be4ce7c7d143f14616a9dc/FAMILY 2014-08-24 06:15:45,617 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testForceAssignWhileClosing/c5ff407162be4ce7c7d143f14616a9dc 2014-08-24 06:15:45,618 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(740): Onlined c5ff407162be4ce7c7d143f14616a9dc; next sequenceid=1 2014-08-24 06:15:45,618 INFO [PostOpenDeployTasks:c5ff407162be4ce7c7d143f14616a9dc] regionserver.HRegionServer(1796): Post open deploy tasks for region=testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,619 DEBUG [FifoRpcScheduler.handler7-thread-4] master.AssignmentManager(4068): Got transition OPENED for {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_OPEN, ts=1408860945608, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,619 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(802): Transition {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_OPEN, ts=1408860945608, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {c5ff407162be4ce7c7d143f14616a9dc state=OPEN, ts=1408860945619, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,619 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStateStore(183): Updating row testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,621 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(359): Onlined c5ff407162be4ce7c7d143f14616a9dc on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,623 INFO [PostOpenDeployTasks:c5ff407162be4ce7c7d143f14616a9dc] regionserver.HRegionServer(1827): Finished post open deploy task for testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,623 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-0] handler.OpenRegionHandler(185): Opened testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,633 INFO [Thread-871] client.HBaseAdmin$5(917): Started disable of testForceAssignWhileClosing 2014-08-24 06:15:45,634 INFO [FifoRpcScheduler.handler7-thread-1] master.HMaster(1979): Client=apurtell//10.229.0.82 disable testForceAssignWhileClosing 2014-08-24 06:15:45,635 DEBUG [FifoRpcScheduler.handler7-thread-1] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testForceAssignWhileClosing/write-master:578260000000001 2014-08-24 06:15:45,645 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(130): Attempting to disable table testForceAssignWhileClosing 2014-08-24 06:15:45,646 INFO [Thread-871] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:45,646 DEBUG [Thread-871] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@7462f92 2014-08-24 06:15:45,646 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(174): Offlining 2 regions. 2014-08-24 06:15:45,647 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2420): Starting unassign of testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. (offlining), current state: {d6c1e0cbf48826990b4d1ae53a5dc74b state=OPEN, ts=1408860945373, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:45,647 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(802): Transition {d6c1e0cbf48826990b4d1ae53a5dc74b state=OPEN, ts=1408860945373, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {d6c1e0cbf48826990b4d1ae53a5dc74b state=PENDING_CLOSE, ts=1408860945647, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:45,647 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.AssignmentManager(2420): Starting unassign of testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. (offlining), current state: {c5ff407162be4ce7c7d143f14616a9dc state=OPEN, ts=1408860945619, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,647 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStateStore(183): Updating row testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. with state=PENDING_CLOSE 2014-08-24 06:15:45,647 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.RegionStates(802): Transition {c5ff407162be4ce7c7d143f14616a9dc state=OPEN, ts=1408860945619, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_CLOSE, ts=1408860945647, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,647 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.RegionStateStore(183): Updating row testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. with state=PENDING_CLOSE 2014-08-24 06:15:45,647 DEBUG [Thread-871-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:45,648 DEBUG [Thread-871] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:45,649 DEBUG [Thread-871-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d003d connected 2014-08-24 06:15:45,650 INFO [PriorityRpcServer.handler=5,queue=0,port=55743] regionserver.HRegionServer(3859): Close d6c1e0cbf48826990b4d1ae53a5dc74b, via zk=no, znode version=-1, on null 2014-08-24 06:15:45,650 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-2] handler.CloseRegionHandler(128): Processing close of testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. 2014-08-24 06:15:45,650 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(1109): Closing testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b.: disabling compactions & flushes 2014-08-24 06:15:45,650 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 for region testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. 2014-08-24 06:15:45,650 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(1136): Updates disabled for region testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. 2014-08-24 06:15:45,651 INFO [StoreCloserThread-testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:45,651 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-2] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:45,651 INFO [RS_CLOSE_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(1218): Closed testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. 2014-08-24 06:15:45,651 DEBUG [FifoRpcScheduler.handler7-thread-2] master.AssignmentManager(4068): Got transition CLOSED for {d6c1e0cbf48826990b4d1ae53a5dc74b state=PENDING_CLOSE, ts=1408860945647, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,652 INFO [PriorityRpcServer.handler=1,queue=0,port=58042] regionserver.HRegionServer(3859): Close c5ff407162be4ce7c7d143f14616a9dc, via zk=no, znode version=-1, on null 2014-08-24 06:15:45,652 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] handler.CloseRegionHandler(128): Processing close of testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,652 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 for region testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,652 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(1109): Closing testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc.: disabling compactions & flushes 2014-08-24 06:15:45,653 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(1136): Updates disabled for region testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,653 INFO [StoreCloserThread-testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:45,653 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:45,653 INFO [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(1218): Closed testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,654 DEBUG [FifoRpcScheduler.handler7-thread-5] master.AssignmentManager(4068): Got transition CLOSED for {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_CLOSE, ts=1408860945647, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,654 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(802): Transition {d6c1e0cbf48826990b4d1ae53a5dc74b state=PENDING_CLOSE, ts=1408860945647, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {d6c1e0cbf48826990b4d1ae53a5dc74b state=OFFLINE, ts=1408860945654, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:45,654 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStateStore(183): Updating row testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. with state=OFFLINE 2014-08-24 06:15:45,655 DEBUG [Thread-871] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@7462f92 2014-08-24 06:15:45,656 DEBUG [Thread-871] client.HBaseAdmin(956): Sleeping= 100ms, waiting for all regions to be disabled in testForceAssignWhileClosing 2014-08-24 06:15:45,656 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(455): Offlined d6c1e0cbf48826990b4d1ae53a5dc74b from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:45,656 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(802): Transition {c5ff407162be4ce7c7d143f14616a9dc state=PENDING_CLOSE, ts=1408860945647, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {c5ff407162be4ce7c7d143f14616a9dc state=OFFLINE, ts=1408860945656, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:45,656 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-2] handler.CloseRegionHandler(182): Closed testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. 2014-08-24 06:15:45,656 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStateStore(183): Updating row testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. with state=OFFLINE 2014-08-24 06:15:45,658 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(455): Offlined c5ff407162be4ce7c7d143f14616a9dc from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:45,659 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] handler.CloseRegionHandler(182): Closed testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. 2014-08-24 06:15:45,697 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler$BulkDisabler(230): Disable waiting until done; 300000 ms remaining; [] 2014-08-24 06:15:45,698 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(190): Disabled table, testForceAssignWhileClosing, is done=true 2014-08-24 06:15:45,700 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testForceAssignWhileClosing/write-master:578260000000001 2014-08-24 06:15:45,757 INFO [Thread-871] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:45,757 DEBUG [Thread-871] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@5c0138f9 2014-08-24 06:15:45,758 DEBUG [Thread-871-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:45,759 DEBUG [Thread-871] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:45,760 DEBUG [Thread-871-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d003e connected 2014-08-24 06:15:45,764 DEBUG [Thread-871] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@5c0138f9 2014-08-24 06:15:45,766 INFO [Thread-871] client.HBaseAdmin(971): Disabled testForceAssignWhileClosing 2014-08-24 06:15:45,766 INFO [FifoRpcScheduler.handler7-thread-3] master.HMaster(1816): Client=apurtell//10.229.0.82 delete testForceAssignWhileClosing 2014-08-24 06:15:45,768 DEBUG [FifoRpcScheduler.handler7-thread-3] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testForceAssignWhileClosing/write-master:578260000000002 2014-08-24 06:15:45,776 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.TableEventHandler(123): Handling table operation C_M_DELETE_TABLE on table testForceAssignWhileClosing 2014-08-24 06:15:45,781 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(93): Deleting regions from META 2014-08-24 06:15:45,783 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] catalog.MetaEditor(495): Deleted [{ENCODED => d6c1e0cbf48826990b4d1ae53a5dc74b, NAME => 'testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b.', STARTKEY => '', ENDKEY => ''}, {ENCODED => c5ff407162be4ce7c7d143f14616a9dc, NAME => 'testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc.', STARTKEY => 'A', ENDKEY => 'Z'}] 2014-08-24 06:15:45,785 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testForceAssignWhileClosing,,1408860945325.d6c1e0cbf48826990b4d1ae53a5dc74b. from FS 2014-08-24 06:15:45,785 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testForceAssignWhileClosing/d6c1e0cbf48826990b4d1ae53a5dc74b 2014-08-24 06:15:45,787 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testForceAssignWhileClosing/d6c1e0cbf48826990b4d1ae53a5dc74b/FAMILY] 2014-08-24 06:15:45,790 INFO [IPC Server handler 5 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741885_1061 127.0.0.1:60268 127.0.0.1:45654 127.0.0.1:39258 2014-08-24 06:15:45,790 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testForceAssignWhileClosing/d6c1e0cbf48826990b4d1ae53a5dc74b 2014-08-24 06:15:45,790 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testForceAssignWhileClosing,A,1408860945457.c5ff407162be4ce7c7d143f14616a9dc. from FS 2014-08-24 06:15:45,790 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testForceAssignWhileClosing/c5ff407162be4ce7c7d143f14616a9dc 2014-08-24 06:15:45,792 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testForceAssignWhileClosing/c5ff407162be4ce7c7d143f14616a9dc/FAMILY] 2014-08-24 06:15:45,795 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741886_1062 127.0.0.1:42916 127.0.0.1:39258 127.0.0.1:45654 2014-08-24 06:15:45,795 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testForceAssignWhileClosing/c5ff407162be4ce7c7d143f14616a9dc 2014-08-24 06:15:45,796 INFO [IPC Server handler 5 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741884_1060 127.0.0.1:39258 127.0.0.1:60268 127.0.0.1:42916 2014-08-24 06:15:45,796 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(114): Table 'testForceAssignWhileClosing' archived! 2014-08-24 06:15:45,796 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(117): Removing 'testForceAssignWhileClosing' descriptor. 2014-08-24 06:15:45,796 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(121): Removing 'testForceAssignWhileClosing' from region states. 2014-08-24 06:15:45,797 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(125): Marking 'testForceAssignWhileClosing' as deleted. 2014-08-24 06:15:45,799 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testForceAssignWhileClosing/write-master:578260000000002 2014-08-24 06:15:45,880 INFO [Thread-871] client.HBaseAdmin(705): Deleted testForceAssignWhileClosing 2014-08-24 06:15:45,892 INFO [main] hbase.ResourceChecker(171): after: master.TestZKLessAMOnCluster#testForceAssignWhileClosing Thread=488 (was 486) - Thread LEAK? -, OpenFileDescriptor=703 (was 702) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=65536 (was 65536), SystemLoadAverage=39 (was 39), ProcessCount=152 (was 152), AvailableMemoryMB=27861 (was 27863), ConnectionCount=9 (was 9) 2014-08-24 06:15:45,904 INFO [main] hbase.ResourceChecker(147): before: master.TestZKLessAMOnCluster#testReportRegionStateTransition Thread=488, OpenFileDescriptor=703, MaxFileDescriptor=65536, SystemLoadAverage=39, ProcessCount=152, AvailableMemoryMB=27861, ConnectionCount=9 2014-08-24 06:15:45,906 INFO [FifoRpcScheduler.handler7-thread-4] master.HMaster(1747): Client=apurtell//10.229.0.82 create 'testReportRegionStateTransition', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2014-08-24 06:15:45,910 DEBUG [FifoRpcScheduler.handler7-thread-4] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testReportRegionStateTransition/write-master:578260000000000 2014-08-24 06:15:45,915 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.CreateTableHandler(148): Create table testReportRegionStateTransition 2014-08-24 06:15:45,923 INFO [IPC Server handler 5 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741887_1063{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:45,923 INFO [IPC Server handler 8 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741887_1063{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:45,924 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741887_1063 size 313 2014-08-24 06:15:45,925 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] util.FSTableDescriptors(640): Wrote descriptor into: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testReportRegionStateTransition/.tabledesc/.tableinfo.0000000001 2014-08-24 06:15:45,926 INFO [RegionOpenAndInitThread-testReportRegionStateTransition-1] regionserver.HRegion(4284): creating HRegion testReportRegionStateTransition HTD == 'testReportRegionStateTransition', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testReportRegionStateTransition 2014-08-24 06:15:45,932 INFO [IPC Server handler 9 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741888_1064{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:45,933 INFO [IPC Server handler 2 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741888_1064{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:45,933 INFO [IPC Server handler 3 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741888_1064 size 64 2014-08-24 06:15:45,934 DEBUG [RegionOpenAndInitThread-testReportRegionStateTransition-1] regionserver.HRegion(639): Instantiated testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. 2014-08-24 06:15:45,934 DEBUG [RegionOpenAndInitThread-testReportRegionStateTransition-1] regionserver.HRegion(1109): Closing testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9.: disabling compactions & flushes 2014-08-24 06:15:45,934 DEBUG [RegionOpenAndInitThread-testReportRegionStateTransition-1] regionserver.HRegion(1136): Updates disabled for region testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. 2014-08-24 06:15:45,934 INFO [RegionOpenAndInitThread-testReportRegionStateTransition-1] regionserver.HRegion(1218): Closed testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. 2014-08-24 06:15:45,937 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] catalog.MetaEditor(278): Added 1 2014-08-24 06:15:45,937 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.AssignmentManager(2696): Bulk assigning 1 region(s) across 3 server(s), round-robin=true 2014-08-24 06:15:45,938 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.AssignmentManager(1597): Assigning 1 region(s) to ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,938 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.GeneralBulkAssigner(177): Timeout-on-RIT=151000 2014-08-24 06:15:45,938 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStates(802): Transition {91ac31b221236a959903ba25f57140a9 state=OFFLINE, ts=1408860945937, server=null} to {91ac31b221236a959903ba25f57140a9 state=PENDING_OPEN, ts=1408860945938, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:45,938 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStateStore(183): Updating row testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,940 INFO [PriorityRpcServer.handler=9,queue=0,port=51857] regionserver.HRegionServer(3718): Open testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. 2014-08-24 06:15:45,943 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.AssignmentManager(1777): Bulk assigning done for ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,943 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(4470): Opening region: {ENCODED => 91ac31b221236a959903ba25f57140a9, NAME => 'testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9.', STARTKEY => '', ENDKEY => ''} 2014-08-24 06:15:45,943 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.GeneralBulkAssigner(153): bulk assigning total 1 regions to 3 servers, took 5ms, with 1 regions still in transition 2014-08-24 06:15:45,944 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.AssignmentManager(2703): Bulk assigning done 2014-08-24 06:15:45,944 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-0] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:45,944 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testReportRegionStateTransition 91ac31b221236a959903ba25f57140a9 2014-08-24 06:15:45,944 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(639): Instantiated testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. 2014-08-24 06:15:45,946 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testReportRegionStateTransition/write-master:578260000000000 2014-08-24 06:15:45,946 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.CreateTableHandler(173): failed. null 2014-08-24 06:15:45,947 INFO [StoreOpener-91ac31b221236a959903ba25f57140a9-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:45,948 DEBUG [StoreOpener-91ac31b221236a959903ba25f57140a9-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testReportRegionStateTransition/91ac31b221236a959903ba25f57140a9/FAMILY 2014-08-24 06:15:45,948 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testReportRegionStateTransition/91ac31b221236a959903ba25f57140a9 2014-08-24 06:15:45,949 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(740): Onlined 91ac31b221236a959903ba25f57140a9; next sequenceid=1 2014-08-24 06:15:45,950 INFO [PostOpenDeployTasks:91ac31b221236a959903ba25f57140a9] regionserver.HRegionServer(1796): Post open deploy tasks for region=testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. 2014-08-24 06:15:45,950 DEBUG [FifoRpcScheduler.handler7-thread-3] master.AssignmentManager(4068): Got transition OPENED for {91ac31b221236a959903ba25f57140a9 state=PENDING_OPEN, ts=1408860945938, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,950 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(802): Transition {91ac31b221236a959903ba25f57140a9 state=PENDING_OPEN, ts=1408860945938, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {91ac31b221236a959903ba25f57140a9 state=OPEN, ts=1408860945950, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:45,950 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStateStore(183): Updating row testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,952 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(359): Onlined 91ac31b221236a959903ba25f57140a9 on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,955 DEBUG [FifoRpcScheduler.handler7-thread-1] master.AssignmentManager(4068): Got transition OPENED for {91ac31b221236a959903ba25f57140a9 state=OPEN, ts=1408860945950, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,955 INFO [FifoRpcScheduler.handler7-thread-1] master.AssignmentManager(4076): Region 91ac31b221236a959903ba25f57140a9 is already OPEN on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:45,955 INFO [PostOpenDeployTasks:91ac31b221236a959903ba25f57140a9] regionserver.HRegionServer(1827): Finished post open deploy task for testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. 2014-08-24 06:15:45,955 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-0] handler.OpenRegionHandler(185): Opened testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,021 INFO [Thread-897] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:46,021 DEBUG [Thread-897] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@5fb32263 2014-08-24 06:15:46,023 DEBUG [Thread-897-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:46,023 DEBUG [Thread-897] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:46,024 DEBUG [Thread-897-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d003f connected 2014-08-24 06:15:46,026 DEBUG [Thread-897] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@5fb32263 2014-08-24 06:15:46,029 DEBUG [Thread-897] catalog.MetaEditor(238): Added testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. 2014-08-24 06:15:46,029 WARN [Thread-897] master.AssignmentManager(1918): Assigning a region not in region states: {ENCODED => a8d26b6ef455fae21d1058eb3642ff0e, NAME => 'testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e.', STARTKEY => 'A', ENDKEY => 'Z'} 2014-08-24 06:15:46,029 DEBUG [Thread-897] master.AssignmentManager(2372): No previous transition plan found (or ignoring an existing plan) for testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e.; generated random plan=hri=testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e., src=, dest=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608; 3 (online=3, available=3) available servers, forceNewPlan=false 2014-08-24 06:15:46,029 INFO [Thread-897] master.AssignmentManager(2083): Assigning testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. to ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,030 INFO [Thread-897] master.RegionStates(802): Transition {a8d26b6ef455fae21d1058eb3642ff0e state=OFFLINE, ts=1408860946029, server=null} to {a8d26b6ef455fae21d1058eb3642ff0e state=PENDING_OPEN, ts=1408860946030, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:46,030 INFO [Thread-897] master.RegionStateStore(183): Updating row testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,032 INFO [PriorityRpcServer.handler=6,queue=0,port=55743] regionserver.HRegionServer(3718): Open testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. 2014-08-24 06:15:46,035 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(4470): Opening region: {ENCODED => a8d26b6ef455fae21d1058eb3642ff0e, NAME => 'testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e.', STARTKEY => 'A', ENDKEY => 'Z'} 2014-08-24 06:15:46,036 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-0] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:46,036 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testReportRegionStateTransition a8d26b6ef455fae21d1058eb3642ff0e 2014-08-24 06:15:46,036 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(639): Instantiated testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. 2014-08-24 06:15:46,037 WARN [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.HRegionFileSystem(788): .regioninfo file not found for region: a8d26b6ef455fae21d1058eb3642ff0e 2014-08-24 06:15:46,043 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741889_1065{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:46,043 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741889_1065{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:46,044 INFO [IPC Server handler 8 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741889_1065 size 66 2014-08-24 06:15:46,047 INFO [StoreOpener-a8d26b6ef455fae21d1058eb3642ff0e-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:46,048 DEBUG [StoreOpener-a8d26b6ef455fae21d1058eb3642ff0e-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testReportRegionStateTransition/a8d26b6ef455fae21d1058eb3642ff0e/FAMILY 2014-08-24 06:15:46,049 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testReportRegionStateTransition/a8d26b6ef455fae21d1058eb3642ff0e 2014-08-24 06:15:46,050 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(740): Onlined a8d26b6ef455fae21d1058eb3642ff0e; next sequenceid=1 2014-08-24 06:15:46,050 INFO [PostOpenDeployTasks:a8d26b6ef455fae21d1058eb3642ff0e] regionserver.HRegionServer(1796): Post open deploy tasks for region=testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. 2014-08-24 06:15:46,051 DEBUG [FifoRpcScheduler.handler7-thread-2] master.AssignmentManager(4068): Got transition OPENED for {a8d26b6ef455fae21d1058eb3642ff0e state=PENDING_OPEN, ts=1408860946030, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,051 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(802): Transition {a8d26b6ef455fae21d1058eb3642ff0e state=PENDING_OPEN, ts=1408860946030, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {a8d26b6ef455fae21d1058eb3642ff0e state=OPEN, ts=1408860946051, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:46,051 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStateStore(183): Updating row testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,053 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(359): Onlined a8d26b6ef455fae21d1058eb3642ff0e on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,055 DEBUG [FifoRpcScheduler.handler7-thread-5] master.AssignmentManager(4068): Got transition OPENED for {a8d26b6ef455fae21d1058eb3642ff0e state=OPEN, ts=1408860946051, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,055 INFO [FifoRpcScheduler.handler7-thread-5] master.AssignmentManager(4076): Region a8d26b6ef455fae21d1058eb3642ff0e is already OPEN on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,055 INFO [PostOpenDeployTasks:a8d26b6ef455fae21d1058eb3642ff0e] regionserver.HRegionServer(1827): Finished post open deploy task for testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. 2014-08-24 06:15:46,055 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-0] handler.OpenRegionHandler(185): Opened testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,064 INFO [Thread-897] client.HBaseAdmin$5(917): Started disable of testReportRegionStateTransition 2014-08-24 06:15:46,065 INFO [FifoRpcScheduler.handler7-thread-3] master.HMaster(1979): Client=apurtell//10.229.0.82 disable testReportRegionStateTransition 2014-08-24 06:15:46,067 DEBUG [FifoRpcScheduler.handler7-thread-3] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testReportRegionStateTransition/write-master:578260000000001 2014-08-24 06:15:46,071 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(130): Attempting to disable table testReportRegionStateTransition 2014-08-24 06:15:46,071 INFO [Thread-897] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:46,072 DEBUG [Thread-897] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@1c7ab4e1 2014-08-24 06:15:46,072 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(174): Offlining 2 regions. 2014-08-24 06:15:46,072 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2420): Starting unassign of testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. (offlining), current state: {91ac31b221236a959903ba25f57140a9 state=OPEN, ts=1408860945950, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:46,072 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.AssignmentManager(2420): Starting unassign of testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. (offlining), current state: {a8d26b6ef455fae21d1058eb3642ff0e state=OPEN, ts=1408860946051, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:46,072 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(802): Transition {91ac31b221236a959903ba25f57140a9 state=OPEN, ts=1408860945950, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {91ac31b221236a959903ba25f57140a9 state=PENDING_CLOSE, ts=1408860946072, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:46,073 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.RegionStates(802): Transition {a8d26b6ef455fae21d1058eb3642ff0e state=OPEN, ts=1408860946051, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {a8d26b6ef455fae21d1058eb3642ff0e state=PENDING_CLOSE, ts=1408860946073, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:46,073 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStateStore(183): Updating row testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. with state=PENDING_CLOSE 2014-08-24 06:15:46,073 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.RegionStateStore(183): Updating row testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. with state=PENDING_CLOSE 2014-08-24 06:15:46,073 DEBUG [Thread-897-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:46,073 DEBUG [Thread-897] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:46,074 DEBUG [Thread-897-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d0040 connected 2014-08-24 06:15:46,075 INFO [PriorityRpcServer.handler=2,queue=0,port=51857] regionserver.HRegionServer(3859): Close 91ac31b221236a959903ba25f57140a9, via zk=no, znode version=-1, on null 2014-08-24 06:15:46,075 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] handler.CloseRegionHandler(128): Processing close of testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. 2014-08-24 06:15:46,075 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 for region testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. 2014-08-24 06:15:46,076 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(1109): Closing testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9.: disabling compactions & flushes 2014-08-24 06:15:46,076 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(1136): Updates disabled for region testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. 2014-08-24 06:15:46,076 INFO [StoreCloserThread-testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:46,076 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:46,077 INFO [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(1218): Closed testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. 2014-08-24 06:15:46,077 DEBUG [Thread-897] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@1c7ab4e1 2014-08-24 06:15:46,077 DEBUG [FifoRpcScheduler.handler7-thread-1] master.AssignmentManager(4068): Got transition CLOSED for {91ac31b221236a959903ba25f57140a9 state=PENDING_CLOSE, ts=1408860946072, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,077 INFO [PriorityRpcServer.handler=9,queue=0,port=55743] regionserver.HRegionServer(3859): Close a8d26b6ef455fae21d1058eb3642ff0e, via zk=no, znode version=-1, on null 2014-08-24 06:15:46,077 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] handler.CloseRegionHandler(128): Processing close of testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. 2014-08-24 06:15:46,078 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 for region testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. 2014-08-24 06:15:46,078 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(1109): Closing testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e.: disabling compactions & flushes 2014-08-24 06:15:46,078 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(1136): Updates disabled for region testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. 2014-08-24 06:15:46,078 INFO [StoreCloserThread-testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:46,078 DEBUG [Thread-897] client.HBaseAdmin(956): Sleeping= 100ms, waiting for all regions to be disabled in testReportRegionStateTransition 2014-08-24 06:15:46,079 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:46,079 INFO [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(1218): Closed testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. 2014-08-24 06:15:46,079 DEBUG [FifoRpcScheduler.handler7-thread-2] master.AssignmentManager(4068): Got transition CLOSED for {a8d26b6ef455fae21d1058eb3642ff0e state=PENDING_CLOSE, ts=1408860946073, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,080 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(802): Transition {91ac31b221236a959903ba25f57140a9 state=PENDING_CLOSE, ts=1408860946072, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {91ac31b221236a959903ba25f57140a9 state=OFFLINE, ts=1408860946080, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:46,080 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStateStore(183): Updating row testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. with state=OFFLINE 2014-08-24 06:15:46,081 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(802): Transition {a8d26b6ef455fae21d1058eb3642ff0e state=PENDING_CLOSE, ts=1408860946073, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {a8d26b6ef455fae21d1058eb3642ff0e state=OFFLINE, ts=1408860946081, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:46,081 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStateStore(183): Updating row testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. with state=OFFLINE 2014-08-24 06:15:46,082 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(455): Offlined 91ac31b221236a959903ba25f57140a9 from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,082 DEBUG [FifoRpcScheduler.handler7-thread-5] master.AssignmentManager(4068): Got transition CLOSED for {91ac31b221236a959903ba25f57140a9 state=OFFLINE, ts=1408860946080, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,082 ERROR [FifoRpcScheduler.handler7-thread-5] master.AssignmentManager(4132): Failed to transtion region from {91ac31b221236a959903ba25f57140a9 state=OFFLINE, ts=1408860946080, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to CLOSED by ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648: 91ac31b221236a959903ba25f57140a9 is not pending close on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,083 INFO [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] regionserver.HRegionServer(1992): Failed to transition {ENCODED => 91ac31b221236a959903ba25f57140a9, NAME => 'testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9.', STARTKEY => '', ENDKEY => ''} to CLOSED: 91ac31b221236a959903ba25f57140a9 is not pending close on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,083 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] handler.CloseRegionHandler(182): Closed testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. 2014-08-24 06:15:46,084 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(455): Offlined a8d26b6ef455fae21d1058eb3642ff0e from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,084 DEBUG [FifoRpcScheduler.handler7-thread-4] master.AssignmentManager(4068): Got transition CLOSED for {a8d26b6ef455fae21d1058eb3642ff0e state=OFFLINE, ts=1408860946081, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,084 ERROR [FifoRpcScheduler.handler7-thread-4] master.AssignmentManager(4132): Failed to transtion region from {a8d26b6ef455fae21d1058eb3642ff0e state=OFFLINE, ts=1408860946081, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to CLOSED by ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608: a8d26b6ef455fae21d1058eb3642ff0e is not pending close on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,084 INFO [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] regionserver.HRegionServer(1992): Failed to transition {ENCODED => a8d26b6ef455fae21d1058eb3642ff0e, NAME => 'testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e.', STARTKEY => 'A', ENDKEY => 'Z'} to CLOSED: a8d26b6ef455fae21d1058eb3642ff0e is not pending close on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,084 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] handler.CloseRegionHandler(182): Closed testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. 2014-08-24 06:15:46,123 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler$BulkDisabler(230): Disable waiting until done; 300000 ms remaining; [] 2014-08-24 06:15:46,124 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(190): Disabled table, testReportRegionStateTransition, is done=true 2014-08-24 06:15:46,125 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testReportRegionStateTransition/write-master:578260000000001 2014-08-24 06:15:46,179 INFO [Thread-897] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:46,179 DEBUG [Thread-897] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4602c19a 2014-08-24 06:15:46,181 DEBUG [Thread-897-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:46,181 DEBUG [Thread-897] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:46,182 DEBUG [Thread-897-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d0041 connected 2014-08-24 06:15:46,184 DEBUG [Thread-897] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4602c19a 2014-08-24 06:15:46,185 INFO [Thread-897] client.HBaseAdmin(971): Disabled testReportRegionStateTransition 2014-08-24 06:15:46,186 INFO [Thread-897] client.HBaseAdmin$5(917): Started disable of testReportRegionStateTransition 2014-08-24 06:15:46,187 INFO [FifoRpcScheduler.handler7-thread-1] master.HMaster(1979): Client=apurtell//10.229.0.82 disable testReportRegionStateTransition 2014-08-24 06:15:46,188 DEBUG [FifoRpcScheduler.handler7-thread-1] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testReportRegionStateTransition/write-master:578260000000002 2014-08-24 06:15:46,191 INFO [FifoRpcScheduler.handler7-thread-1] handler.DisableTableHandler(99): Table testReportRegionStateTransition isn't enabled; skipping disable 2014-08-24 06:15:46,192 DEBUG [FifoRpcScheduler.handler7-thread-1] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testReportRegionStateTransition/write-master:578260000000002 2014-08-24 06:15:46,193 DEBUG [Thread-897] hbase.HBaseTestingUtility(1493): Table: testReportRegionStateTransition already disabled, so just deleting it. 2014-08-24 06:15:46,193 INFO [FifoRpcScheduler.handler7-thread-2] master.HMaster(1816): Client=apurtell//10.229.0.82 delete testReportRegionStateTransition 2014-08-24 06:15:46,198 DEBUG [FifoRpcScheduler.handler7-thread-2] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testReportRegionStateTransition/write-master:578260000000003 2014-08-24 06:15:46,203 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.TableEventHandler(123): Handling table operation C_M_DELETE_TABLE on table testReportRegionStateTransition 2014-08-24 06:15:46,206 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(93): Deleting regions from META 2014-08-24 06:15:46,209 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] catalog.MetaEditor(495): Deleted [{ENCODED => 91ac31b221236a959903ba25f57140a9, NAME => 'testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9.', STARTKEY => '', ENDKEY => ''}, {ENCODED => a8d26b6ef455fae21d1058eb3642ff0e, NAME => 'testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e.', STARTKEY => 'A', ENDKEY => 'Z'}] 2014-08-24 06:15:46,210 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testReportRegionStateTransition,,1408860945906.91ac31b221236a959903ba25f57140a9. from FS 2014-08-24 06:15:46,210 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testReportRegionStateTransition/91ac31b221236a959903ba25f57140a9 2014-08-24 06:15:46,212 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testReportRegionStateTransition/91ac31b221236a959903ba25f57140a9/FAMILY] 2014-08-24 06:15:46,215 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741888_1064 127.0.0.1:42916 127.0.0.1:60268 127.0.0.1:39258 2014-08-24 06:15:46,215 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testReportRegionStateTransition/91ac31b221236a959903ba25f57140a9 2014-08-24 06:15:46,215 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testReportRegionStateTransition,A,1408860946027.a8d26b6ef455fae21d1058eb3642ff0e. from FS 2014-08-24 06:15:46,215 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testReportRegionStateTransition/a8d26b6ef455fae21d1058eb3642ff0e 2014-08-24 06:15:46,217 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testReportRegionStateTransition/a8d26b6ef455fae21d1058eb3642ff0e/FAMILY] 2014-08-24 06:15:46,220 INFO [IPC Server handler 1 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741889_1065 127.0.0.1:39258 127.0.0.1:42916 127.0.0.1:60268 2014-08-24 06:15:46,220 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testReportRegionStateTransition/a8d26b6ef455fae21d1058eb3642ff0e 2014-08-24 06:15:46,220 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741887_1063 127.0.0.1:39258 127.0.0.1:42916 127.0.0.1:45654 2014-08-24 06:15:46,221 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(114): Table 'testReportRegionStateTransition' archived! 2014-08-24 06:15:46,221 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(117): Removing 'testReportRegionStateTransition' descriptor. 2014-08-24 06:15:46,221 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(121): Removing 'testReportRegionStateTransition' from region states. 2014-08-24 06:15:46,221 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(125): Marking 'testReportRegionStateTransition' as deleted. 2014-08-24 06:15:46,224 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testReportRegionStateTransition/write-master:578260000000003 2014-08-24 06:15:46,307 INFO [Thread-897] client.HBaseAdmin(705): Deleted testReportRegionStateTransition 2014-08-24 06:15:46,320 INFO [main] hbase.ResourceChecker(171): after: master.TestZKLessAMOnCluster#testReportRegionStateTransition Thread=488 (was 488), OpenFileDescriptor=702 (was 703), MaxFileDescriptor=65536 (was 65536), SystemLoadAverage=39 (was 39), ProcessCount=152 (was 152), AvailableMemoryMB=27861 (was 27861), ConnectionCount=9 (was 9) 2014-08-24 06:15:46,332 INFO [main] hbase.ResourceChecker(147): before: master.TestZKLessAMOnCluster#testMoveRegion Thread=488, OpenFileDescriptor=702, MaxFileDescriptor=65536, SystemLoadAverage=39, ProcessCount=152, AvailableMemoryMB=27860, ConnectionCount=9 2014-08-24 06:15:46,333 INFO [FifoRpcScheduler.handler7-thread-5] master.HMaster(1747): Client=apurtell//10.229.0.82 create 'testMoveRegion', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2014-08-24 06:15:46,338 DEBUG [FifoRpcScheduler.handler7-thread-5] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testMoveRegion/write-master:578260000000000 2014-08-24 06:15:46,344 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.CreateTableHandler(148): Create table testMoveRegion 2014-08-24 06:15:46,352 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741890_1066{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW]]} size 0 2014-08-24 06:15:46,352 INFO [IPC Server handler 9 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741890_1066{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW]]} size 0 2014-08-24 06:15:46,353 INFO [IPC Server handler 7 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741890_1066 size 296 2014-08-24 06:15:46,354 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] util.FSTableDescriptors(640): Wrote descriptor into: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/.tabledesc/.tableinfo.0000000001 2014-08-24 06:15:46,355 INFO [RegionOpenAndInitThread-testMoveRegion-1] regionserver.HRegion(4284): creating HRegion testMoveRegion HTD == 'testMoveRegion', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testMoveRegion 2014-08-24 06:15:46,355 INFO [RegionOpenAndInitThread-testMoveRegion-2] regionserver.HRegion(4284): creating HRegion testMoveRegion HTD == 'testMoveRegion', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testMoveRegion 2014-08-24 06:15:46,355 INFO [RegionOpenAndInitThread-testMoveRegion-3] regionserver.HRegion(4284): creating HRegion testMoveRegion HTD == 'testMoveRegion', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testMoveRegion 2014-08-24 06:15:46,356 INFO [RegionOpenAndInitThread-testMoveRegion-5] regionserver.HRegion(4284): creating HRegion testMoveRegion HTD == 'testMoveRegion', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testMoveRegion 2014-08-24 06:15:46,355 INFO [RegionOpenAndInitThread-testMoveRegion-4] regionserver.HRegion(4284): creating HRegion testMoveRegion HTD == 'testMoveRegion', {NAME => 'FAMILY', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:59365/user/apurtell/hbase/.tmp Table name == testMoveRegion 2014-08-24 06:15:46,364 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741892_1068{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW]]} size 0 2014-08-24 06:15:46,365 INFO [IPC Server handler 5 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741891_1067{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:46,368 INFO [IPC Server handler 8 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741892_1068{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW]]} size 0 2014-08-24 06:15:46,369 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741891_1067{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:46,369 INFO [IPC Server handler 2 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741891_1067{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:46,370 DEBUG [RegionOpenAndInitThread-testMoveRegion-2] regionserver.HRegion(639): Instantiated testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. 2014-08-24 06:15:46,370 DEBUG [RegionOpenAndInitThread-testMoveRegion-2] regionserver.HRegion(1109): Closing testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb.: disabling compactions & flushes 2014-08-24 06:15:46,370 INFO [IPC Server handler 7 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741893_1069{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:46,370 DEBUG [RegionOpenAndInitThread-testMoveRegion-2] regionserver.HRegion(1136): Updates disabled for region testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. 2014-08-24 06:15:46,370 DEBUG [RegionOpenAndInitThread-testMoveRegion-1] regionserver.HRegion(639): Instantiated testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. 2014-08-24 06:15:46,370 INFO [IPC Server handler 3 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741893_1069{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:46,370 DEBUG [RegionOpenAndInitThread-testMoveRegion-1] regionserver.HRegion(1109): Closing testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef.: disabling compactions & flushes 2014-08-24 06:15:46,370 INFO [RegionOpenAndInitThread-testMoveRegion-2] regionserver.HRegion(1218): Closed testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. 2014-08-24 06:15:46,371 DEBUG [RegionOpenAndInitThread-testMoveRegion-1] regionserver.HRegion(1136): Updates disabled for region testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. 2014-08-24 06:15:46,371 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741892_1068 size 49 2014-08-24 06:15:46,371 INFO [RegionOpenAndInitThread-testMoveRegion-1] regionserver.HRegion(1218): Closed testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. 2014-08-24 06:15:46,371 INFO [IPC Server handler 5 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741893_1069 size 49 2014-08-24 06:15:46,374 INFO [IPC Server handler 7 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741894_1070{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:46,374 INFO [IPC Server handler 3 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741895_1071{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:46,374 DEBUG [RegionOpenAndInitThread-testMoveRegion-3] regionserver.HRegion(639): Instantiated testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,374 INFO [IPC Server handler 8 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741894_1070{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:46,374 DEBUG [RegionOpenAndInitThread-testMoveRegion-3] regionserver.HRegion(1109): Closing testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf.: disabling compactions & flushes 2014-08-24 06:15:46,375 DEBUG [RegionOpenAndInitThread-testMoveRegion-3] regionserver.HRegion(1136): Updates disabled for region testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,375 INFO [RegionOpenAndInitThread-testMoveRegion-3] regionserver.HRegion(1218): Closed testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,375 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741894_1070 size 48 2014-08-24 06:15:46,375 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741895_1071{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:46,375 DEBUG [RegionOpenAndInitThread-testMoveRegion-5] regionserver.HRegion(639): Instantiated testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. 2014-08-24 06:15:46,375 DEBUG [RegionOpenAndInitThread-testMoveRegion-5] regionserver.HRegion(1109): Closing testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03.: disabling compactions & flushes 2014-08-24 06:15:46,375 INFO [IPC Server handler 1 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741895_1071 size 49 2014-08-24 06:15:46,375 DEBUG [RegionOpenAndInitThread-testMoveRegion-5] regionserver.HRegion(1136): Updates disabled for region testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. 2014-08-24 06:15:46,375 INFO [RegionOpenAndInitThread-testMoveRegion-5] regionserver.HRegion(1218): Closed testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. 2014-08-24 06:15:46,415 DEBUG [RegionOpenAndInitThread-testMoveRegion-4] regionserver.HRegion(639): Instantiated testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. 2014-08-24 06:15:46,415 DEBUG [RegionOpenAndInitThread-testMoveRegion-4] regionserver.HRegion(1109): Closing testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c.: disabling compactions & flushes 2014-08-24 06:15:46,415 DEBUG [RegionOpenAndInitThread-testMoveRegion-4] regionserver.HRegion(1136): Updates disabled for region testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. 2014-08-24 06:15:46,415 INFO [RegionOpenAndInitThread-testMoveRegion-4] regionserver.HRegion(1218): Closed testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. 2014-08-24 06:15:46,420 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] catalog.MetaEditor(278): Added 5 2014-08-24 06:15:46,420 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.AssignmentManager(2696): Bulk assigning 5 region(s) across 3 server(s), round-robin=true 2014-08-24 06:15:46,420 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.AssignmentManager(1597): Assigning 2 region(s) to ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,421 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.AssignmentManager(1597): Assigning 1 region(s) to ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,421 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStates(802): Transition {98d257181059e18e5d88f247eaefedfb state=OFFLINE, ts=1408860946420, server=null} to {98d257181059e18e5d88f247eaefedfb state=PENDING_OPEN, ts=1408860946421, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:46,421 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.GeneralBulkAssigner(177): Timeout-on-RIT=152000 2014-08-24 06:15:46,421 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStateStore(183): Updating row testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,421 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.RegionStates(802): Transition {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=OFFLINE, ts=1408860946420, server=null} to {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=PENDING_OPEN, ts=1408860946421, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:46,421 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.AssignmentManager(1597): Assigning 2 region(s) to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,421 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.RegionStateStore(183): Updating row testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,421 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.RegionStates(802): Transition {d3f4637d404da158b76d1a44746ca2ef state=OFFLINE, ts=1408860946420, server=null} to {d3f4637d404da158b76d1a44746ca2ef state=PENDING_OPEN, ts=1408860946421, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:46,421 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.RegionStateStore(183): Updating row testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,423 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStates(802): Transition {3ffbbb0df93a5a80e386b6d2ac574e03 state=OFFLINE, ts=1408860946420, server=null} to {3ffbbb0df93a5a80e386b6d2ac574e03 state=PENDING_OPEN, ts=1408860946423, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:46,423 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.RegionStateStore(183): Updating row testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,425 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.RegionStates(802): Transition {28f6d3e65d1c195f5df4cef92084bd8c state=OFFLINE, ts=1408860946420, server=null} to {28f6d3e65d1c195f5df4cef92084bd8c state=PENDING_OPEN, ts=1408860946425, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:46,425 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.RegionStateStore(183): Updating row testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,427 INFO [PriorityRpcServer.handler=1,queue=0,port=55743] regionserver.HRegionServer(3718): Open testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,429 INFO [PriorityRpcServer.handler=2,queue=0,port=58042] regionserver.HRegionServer(3718): Open testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. 2014-08-24 06:15:46,430 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-1] master.AssignmentManager(1777): Bulk assigning done for ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,430 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(4470): Opening region: {ENCODED => ceafd98e56e0ec8e4fb7c47fbbc93bbf, NAME => 'testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf.', STARTKEY => 'I', ENDKEY => 'Q'} 2014-08-24 06:15:46,430 INFO [PriorityRpcServer.handler=4,queue=0,port=51857] regionserver.HRegionServer(3718): Open testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. 2014-08-24 06:15:46,431 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-2] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:46,431 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testMoveRegion ceafd98e56e0ec8e4fb7c47fbbc93bbf 2014-08-24 06:15:46,431 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(639): Instantiated testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,431 INFO [PriorityRpcServer.handler=2,queue=0,port=58042] regionserver.HRegionServer(3718): Open testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. 2014-08-24 06:15:46,432 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(4470): Opening region: {ENCODED => d3f4637d404da158b76d1a44746ca2ef, NAME => 'testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef.', STARTKEY => '', ENDKEY => 'A'} 2014-08-24 06:15:46,432 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-2] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:46,432 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-2] master.AssignmentManager(1777): Bulk assigning done for ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,432 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(4470): Opening region: {ENCODED => 28f6d3e65d1c195f5df4cef92084bd8c, NAME => 'testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c.', STARTKEY => 'Q', ENDKEY => 'Z'} 2014-08-24 06:15:46,432 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testMoveRegion d3f4637d404da158b76d1a44746ca2ef 2014-08-24 06:15:46,432 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-1] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:46,432 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(639): Instantiated testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. 2014-08-24 06:15:46,433 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testMoveRegion 28f6d3e65d1c195f5df4cef92084bd8c 2014-08-24 06:15:46,433 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(639): Instantiated testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. 2014-08-24 06:15:46,434 INFO [PriorityRpcServer.handler=4,queue=0,port=51857] regionserver.HRegionServer(3718): Open testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. 2014-08-24 06:15:46,434 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(4470): Opening region: {ENCODED => 98d257181059e18e5d88f247eaefedfb, NAME => 'testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb.', STARTKEY => 'A', ENDKEY => 'I'} 2014-08-24 06:15:46,434 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-1] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:46,434 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-GeneralBulkAssigner-0] master.AssignmentManager(1777): Bulk assigning done for ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,434 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(4470): Opening region: {ENCODED => 3ffbbb0df93a5a80e386b6d2ac574e03, NAME => 'testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03.', STARTKEY => 'Z', ENDKEY => ''} 2014-08-24 06:15:46,434 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testMoveRegion 98d257181059e18e5d88f247eaefedfb 2014-08-24 06:15:46,434 INFO [StoreOpener-ceafd98e56e0ec8e4fb7c47fbbc93bbf-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:46,434 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.GeneralBulkAssigner(153): bulk assigning total 5 regions to 3 servers, took 13ms, with 5 regions still in transition 2014-08-24 06:15:46,435 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(639): Instantiated testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. 2014-08-24 06:15:46,435 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-2] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:46,435 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] master.AssignmentManager(2703): Bulk assigning done 2014-08-24 06:15:46,435 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testMoveRegion 3ffbbb0df93a5a80e386b6d2ac574e03 2014-08-24 06:15:46,435 DEBUG [StoreOpener-ceafd98e56e0ec8e4fb7c47fbbc93bbf-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testMoveRegion/ceafd98e56e0ec8e4fb7c47fbbc93bbf/FAMILY 2014-08-24 06:15:46,436 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(639): Instantiated testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. 2014-08-24 06:15:46,436 INFO [StoreOpener-d3f4637d404da158b76d1a44746ca2ef-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:46,437 INFO [StoreOpener-28f6d3e65d1c195f5df4cef92084bd8c-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:46,437 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testMoveRegion/ceafd98e56e0ec8e4fb7c47fbbc93bbf 2014-08-24 06:15:46,438 DEBUG [StoreOpener-d3f4637d404da158b76d1a44746ca2ef-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testMoveRegion/d3f4637d404da158b76d1a44746ca2ef/FAMILY 2014-08-24 06:15:46,438 DEBUG [StoreOpener-28f6d3e65d1c195f5df4cef92084bd8c-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testMoveRegion/28f6d3e65d1c195f5df4cef92084bd8c/FAMILY 2014-08-24 06:15:46,438 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testMoveRegion/d3f4637d404da158b76d1a44746ca2ef 2014-08-24 06:15:46,438 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testMoveRegion/write-master:578260000000000 2014-08-24 06:15:46,438 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.CreateTableHandler(173): failed. null 2014-08-24 06:15:46,439 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testMoveRegion/28f6d3e65d1c195f5df4cef92084bd8c 2014-08-24 06:15:46,439 INFO [RS_OPEN_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(740): Onlined ceafd98e56e0ec8e4fb7c47fbbc93bbf; next sequenceid=1 2014-08-24 06:15:46,439 INFO [StoreOpener-98d257181059e18e5d88f247eaefedfb-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:46,439 INFO [PostOpenDeployTasks:ceafd98e56e0ec8e4fb7c47fbbc93bbf] regionserver.HRegionServer(1796): Post open deploy tasks for region=testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,440 INFO [StoreOpener-3ffbbb0df93a5a80e386b6d2ac574e03-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:46,440 DEBUG [StoreOpener-98d257181059e18e5d88f247eaefedfb-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testMoveRegion/98d257181059e18e5d88f247eaefedfb/FAMILY 2014-08-24 06:15:46,441 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(740): Onlined 28f6d3e65d1c195f5df4cef92084bd8c; next sequenceid=1 2014-08-24 06:15:46,440 DEBUG [FifoRpcScheduler.handler7-thread-1] master.AssignmentManager(4068): Got transition OPENED for {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=PENDING_OPEN, ts=1408860946421, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,440 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(740): Onlined d3f4637d404da158b76d1a44746ca2ef; next sequenceid=1 2014-08-24 06:15:46,441 INFO [PostOpenDeployTasks:28f6d3e65d1c195f5df4cef92084bd8c] regionserver.HRegionServer(1796): Post open deploy tasks for region=testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. 2014-08-24 06:15:46,441 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(802): Transition {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=PENDING_OPEN, ts=1408860946421, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=OPEN, ts=1408860946441, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:46,443 DEBUG [StoreOpener-3ffbbb0df93a5a80e386b6d2ac574e03-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testMoveRegion/3ffbbb0df93a5a80e386b6d2ac574e03/FAMILY 2014-08-24 06:15:46,444 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStateStore(183): Updating row testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,444 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testMoveRegion/98d257181059e18e5d88f247eaefedfb 2014-08-24 06:15:46,444 INFO [PostOpenDeployTasks:d3f4637d404da158b76d1a44746ca2ef] regionserver.HRegionServer(1796): Post open deploy tasks for region=testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. 2014-08-24 06:15:46,444 DEBUG [FifoRpcScheduler.handler7-thread-5] master.AssignmentManager(4068): Got transition OPENED for {28f6d3e65d1c195f5df4cef92084bd8c state=PENDING_OPEN, ts=1408860946425, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,444 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(802): Transition {28f6d3e65d1c195f5df4cef92084bd8c state=PENDING_OPEN, ts=1408860946425, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {28f6d3e65d1c195f5df4cef92084bd8c state=OPEN, ts=1408860946444, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:46,444 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStateStore(183): Updating row testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,444 DEBUG [FifoRpcScheduler.handler7-thread-2] master.AssignmentManager(4068): Got transition OPENED for {d3f4637d404da158b76d1a44746ca2ef state=PENDING_OPEN, ts=1408860946421, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,444 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(802): Transition {d3f4637d404da158b76d1a44746ca2ef state=PENDING_OPEN, ts=1408860946421, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {d3f4637d404da158b76d1a44746ca2ef state=OPEN, ts=1408860946444, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:46,445 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStateStore(183): Updating row testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,445 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testMoveRegion/3ffbbb0df93a5a80e386b6d2ac574e03 2014-08-24 06:15:46,445 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(740): Onlined 98d257181059e18e5d88f247eaefedfb; next sequenceid=1 2014-08-24 06:15:46,446 INFO [PostOpenDeployTasks:98d257181059e18e5d88f247eaefedfb] regionserver.HRegionServer(1796): Post open deploy tasks for region=testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. 2014-08-24 06:15:46,446 DEBUG [FifoRpcScheduler.handler7-thread-4] master.AssignmentManager(4068): Got transition OPENED for {98d257181059e18e5d88f247eaefedfb state=PENDING_OPEN, ts=1408860946421, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,446 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(802): Transition {98d257181059e18e5d88f247eaefedfb state=PENDING_OPEN, ts=1408860946421, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {98d257181059e18e5d88f247eaefedfb state=OPEN, ts=1408860946446, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:46,446 INFO [RS_OPEN_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(740): Onlined 3ffbbb0df93a5a80e386b6d2ac574e03; next sequenceid=1 2014-08-24 06:15:46,446 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStateStore(183): Updating row testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,446 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(359): Onlined ceafd98e56e0ec8e4fb7c47fbbc93bbf on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,446 INFO [PostOpenDeployTasks:3ffbbb0df93a5a80e386b6d2ac574e03] regionserver.HRegionServer(1796): Post open deploy tasks for region=testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. 2014-08-24 06:15:46,447 DEBUG [FifoRpcScheduler.handler7-thread-3] master.AssignmentManager(4068): Got transition OPENED for {3ffbbb0df93a5a80e386b6d2ac574e03 state=PENDING_OPEN, ts=1408860946423, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,447 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(802): Transition {3ffbbb0df93a5a80e386b6d2ac574e03 state=PENDING_OPEN, ts=1408860946423, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {3ffbbb0df93a5a80e386b6d2ac574e03 state=OPEN, ts=1408860946447, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:46,447 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStateStore(183): Updating row testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,448 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(359): Onlined 98d257181059e18e5d88f247eaefedfb on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,449 INFO [PostOpenDeployTasks:ceafd98e56e0ec8e4fb7c47fbbc93bbf] regionserver.HRegionServer(1827): Finished post open deploy task for testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,449 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:55743-2] handler.OpenRegionHandler(185): Opened testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. on ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,450 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(359): Onlined 3ffbbb0df93a5a80e386b6d2ac574e03 on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,451 INFO [PostOpenDeployTasks:98d257181059e18e5d88f247eaefedfb] regionserver.HRegionServer(1827): Finished post open deploy task for testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. 2014-08-24 06:15:46,451 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-1] handler.OpenRegionHandler(185): Opened testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,452 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(359): Onlined d3f4637d404da158b76d1a44746ca2ef on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,453 INFO [PostOpenDeployTasks:3ffbbb0df93a5a80e386b6d2ac574e03] regionserver.HRegionServer(1827): Finished post open deploy task for testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. 2014-08-24 06:15:46,453 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:51857-2] handler.OpenRegionHandler(185): Opened testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. on ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,455 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(359): Onlined 28f6d3e65d1c195f5df4cef92084bd8c on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,455 INFO [PostOpenDeployTasks:d3f4637d404da158b76d1a44746ca2ef] regionserver.HRegionServer(1827): Finished post open deploy task for testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. 2014-08-24 06:15:46,455 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-2] handler.OpenRegionHandler(185): Opened testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,457 INFO [PostOpenDeployTasks:28f6d3e65d1c195f5df4cef92084bd8c] regionserver.HRegionServer(1827): Finished post open deploy task for testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. 2014-08-24 06:15:46,457 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-1] handler.OpenRegionHandler(185): Opened testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,661 INFO [Thread-922] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:46,661 DEBUG [Thread-922] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@502eca9c 2014-08-24 06:15:46,662 DEBUG [Thread-922-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:46,663 DEBUG [Thread-922] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:46,664 DEBUG [Thread-922-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d0042 connected 2014-08-24 06:15:46,667 DEBUG [Thread-922] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@502eca9c 2014-08-24 06:15:46,669 INFO [FifoRpcScheduler.handler7-thread-4] master.HMaster(1717): Client=apurtell//10.229.0.82 move hri=testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf., src=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608, dest=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571, running balancer 2014-08-24 06:15:46,670 DEBUG [FifoRpcScheduler.handler7-thread-4] master.AssignmentManager(2420): Starting unassign of testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. (offlining), current state: {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=OPEN, ts=1408860946441, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:46,670 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(802): Transition {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=OPEN, ts=1408860946441, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=PENDING_CLOSE, ts=1408860946670, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:46,670 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStateStore(183): Updating row testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. with state=PENDING_CLOSE 2014-08-24 06:15:46,672 INFO [PriorityRpcServer.handler=2,queue=0,port=55743] regionserver.HRegionServer(3859): Close ceafd98e56e0ec8e4fb7c47fbbc93bbf, via zk=no, znode version=-1, on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,672 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] handler.CloseRegionHandler(128): Processing close of testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,672 DEBUG [FifoRpcScheduler.handler7-thread-4] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 for region testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,673 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(1109): Closing testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf.: disabling compactions & flushes 2014-08-24 06:15:46,673 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(1136): Updates disabled for region testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,673 INFO [StoreCloserThread-testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:46,673 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:46,674 INFO [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(1218): Closed testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,674 INFO [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] regionserver.HRegionServer(4527): Adding moved region record: ceafd98e56e0ec8e4fb7c47fbbc93bbf to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571:58042 as of 1 2014-08-24 06:15:46,674 DEBUG [FifoRpcScheduler.handler7-thread-3] master.AssignmentManager(4068): Got transition CLOSED for {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=PENDING_CLOSE, ts=1408860946670, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,676 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(802): Transition {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=PENDING_CLOSE, ts=1408860946670, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=CLOSED, ts=1408860946676, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} 2014-08-24 06:15:46,676 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStateStore(183): Updating row testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. with state=CLOSED 2014-08-24 06:15:46,678 DEBUG [AM.-pool37-t24] master.AssignmentManager(2342): Found an existing plan for testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. destination server is ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 accepted as a dest server = true 2014-08-24 06:15:46,678 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] handler.CloseRegionHandler(182): Closed testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,678 DEBUG [AM.-pool37-t24] master.AssignmentManager(2381): Using pre-existing plan for testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf.; plan=hri=testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf., src=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608, dest=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,679 INFO [AM.-pool37-t24] master.AssignmentManager(2083): Assigning testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,679 INFO [AM.-pool37-t24] master.RegionStates(802): Transition {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=CLOSED, ts=1408860946676, server=ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608} to {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=PENDING_OPEN, ts=1408860946679, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:46,679 INFO [AM.-pool37-t24] master.RegionStateStore(183): Updating row testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. with state=PENDING_OPEN&sn=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,681 INFO [PriorityRpcServer.handler=3,queue=0,port=58042] regionserver.HRegionServer(3718): Open testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,683 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(4470): Opening region: {ENCODED => ceafd98e56e0ec8e4fb7c47fbbc93bbf, NAME => 'testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf.', STARTKEY => 'I', ENDKEY => 'Q'} 2014-08-24 06:15:46,683 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-0] coprocessor.CoprocessorHost(169): System coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver was loaded successfully with priority (536870911). 2014-08-24 06:15:46,683 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testMoveRegion ceafd98e56e0ec8e4fb7c47fbbc93bbf 2014-08-24 06:15:46,683 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(639): Instantiated testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,685 INFO [StoreOpener-ceafd98e56e0ec8e4fb7c47fbbc93bbf-1] compactions.CompactionConfiguration(88): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000 2014-08-24 06:15:46,686 DEBUG [StoreOpener-ceafd98e56e0ec8e4fb7c47fbbc93bbf-1] regionserver.HRegionFileSystem(193): No StoreFiles for: hdfs://localhost:59365/user/apurtell/hbase/data/default/testMoveRegion/ceafd98e56e0ec8e4fb7c47fbbc93bbf/FAMILY 2014-08-24 06:15:46,687 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(3107): Found 0 recovered edits file(s) under hdfs://localhost:59365/user/apurtell/hbase/data/default/testMoveRegion/ceafd98e56e0ec8e4fb7c47fbbc93bbf 2014-08-24 06:15:46,688 INFO [RS_OPEN_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(740): Onlined ceafd98e56e0ec8e4fb7c47fbbc93bbf; next sequenceid=1 2014-08-24 06:15:46,688 INFO [PostOpenDeployTasks:ceafd98e56e0ec8e4fb7c47fbbc93bbf] regionserver.HRegionServer(1796): Post open deploy tasks for region=testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,689 DEBUG [FifoRpcScheduler.handler7-thread-2] master.AssignmentManager(4068): Got transition OPENED for {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=PENDING_OPEN, ts=1408860946679, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,689 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(802): Transition {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=PENDING_OPEN, ts=1408860946679, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=OPEN, ts=1408860946689, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:46,689 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStateStore(183): Updating row testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. with state=OPEN&openSeqNum=1&server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,691 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(359): Onlined ceafd98e56e0ec8e4fb7c47fbbc93bbf on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,691 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(367): Offlined ceafd98e56e0ec8e4fb7c47fbbc93bbf from ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:46,693 INFO [PostOpenDeployTasks:ceafd98e56e0ec8e4fb7c47fbbc93bbf] regionserver.HRegionServer(1827): Finished post open deploy task for testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,693 DEBUG [RS_OPEN_REGION-ip-10-229-0-82:58042-0] handler.OpenRegionHandler(185): Opened testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. on ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,703 INFO [Thread-922] client.HBaseAdmin$5(917): Started disable of testMoveRegion 2014-08-24 06:15:46,703 INFO [FifoRpcScheduler.handler7-thread-1] master.HMaster(1979): Client=apurtell//10.229.0.82 disable testMoveRegion 2014-08-24 06:15:46,705 DEBUG [FifoRpcScheduler.handler7-thread-1] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testMoveRegion/write-master:578260000000001 2014-08-24 06:15:46,711 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(130): Attempting to disable table testMoveRegion 2014-08-24 06:15:46,712 INFO [Thread-922] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:46,712 DEBUG [Thread-922] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@627ad5a0 2014-08-24 06:15:46,713 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(174): Offlining 5 regions. 2014-08-24 06:15:46,713 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2420): Starting unassign of testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. (offlining), current state: {d3f4637d404da158b76d1a44746ca2ef state=OPEN, ts=1408860946444, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:46,713 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(802): Transition {d3f4637d404da158b76d1a44746ca2ef state=OPEN, ts=1408860946444, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {d3f4637d404da158b76d1a44746ca2ef state=PENDING_CLOSE, ts=1408860946713, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:46,713 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.AssignmentManager(2420): Starting unassign of testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. (offlining), current state: {98d257181059e18e5d88f247eaefedfb state=OPEN, ts=1408860946446, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:46,713 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStateStore(183): Updating row testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. with state=PENDING_CLOSE 2014-08-24 06:15:46,713 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-2] master.AssignmentManager(2420): Starting unassign of testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. (offlining), current state: {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=OPEN, ts=1408860946689, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:46,714 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-4] master.AssignmentManager(2420): Starting unassign of testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. (offlining), current state: {3ffbbb0df93a5a80e386b6d2ac574e03 state=OPEN, ts=1408860946447, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:46,714 DEBUG [Thread-922-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:46,713 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-3] master.AssignmentManager(2420): Starting unassign of testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. (offlining), current state: {28f6d3e65d1c195f5df4cef92084bd8c state=OPEN, ts=1408860946444, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:46,714 DEBUG [Thread-922] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:46,713 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.RegionStates(802): Transition {98d257181059e18e5d88f247eaefedfb state=OPEN, ts=1408860946446, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {98d257181059e18e5d88f247eaefedfb state=PENDING_CLOSE, ts=1408860946713, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:46,714 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-3] master.RegionStates(802): Transition {28f6d3e65d1c195f5df4cef92084bd8c state=OPEN, ts=1408860946444, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {28f6d3e65d1c195f5df4cef92084bd8c state=PENDING_CLOSE, ts=1408860946714, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:46,714 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-4] master.RegionStates(802): Transition {3ffbbb0df93a5a80e386b6d2ac574e03 state=OPEN, ts=1408860946447, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {3ffbbb0df93a5a80e386b6d2ac574e03 state=PENDING_CLOSE, ts=1408860946714, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:46,714 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-2] master.RegionStates(802): Transition {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=OPEN, ts=1408860946689, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=PENDING_CLOSE, ts=1408860946714, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:46,715 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-4] master.RegionStateStore(183): Updating row testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. with state=PENDING_CLOSE 2014-08-24 06:15:46,714 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-3] master.RegionStateStore(183): Updating row testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. with state=PENDING_CLOSE 2014-08-24 06:15:46,714 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.RegionStateStore(183): Updating row testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. with state=PENDING_CLOSE 2014-08-24 06:15:46,715 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-2] master.RegionStateStore(183): Updating row testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. with state=PENDING_CLOSE 2014-08-24 06:15:46,715 DEBUG [Thread-922-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d0043 connected 2014-08-24 06:15:46,716 INFO [PriorityRpcServer.handler=6,queue=0,port=58042] regionserver.HRegionServer(3859): Close d3f4637d404da158b76d1a44746ca2ef, via zk=no, znode version=-1, on null 2014-08-24 06:15:46,716 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] handler.CloseRegionHandler(128): Processing close of testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. 2014-08-24 06:15:46,716 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 for region testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. 2014-08-24 06:15:46,717 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(1109): Closing testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef.: disabling compactions & flushes 2014-08-24 06:15:46,717 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(1136): Updates disabled for region testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. 2014-08-24 06:15:46,717 INFO [StoreCloserThread-testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:46,717 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:46,717 INFO [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(1218): Closed testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. 2014-08-24 06:15:46,718 DEBUG [FifoRpcScheduler.handler7-thread-4] master.AssignmentManager(4068): Got transition CLOSED for {d3f4637d404da158b76d1a44746ca2ef state=PENDING_CLOSE, ts=1408860946713, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,718 INFO [PriorityRpcServer.handler=7,queue=0,port=58042] regionserver.HRegionServer(3859): Close ceafd98e56e0ec8e4fb7c47fbbc93bbf, via zk=no, znode version=-1, on null 2014-08-24 06:15:46,718 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] handler.CloseRegionHandler(128): Processing close of testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,719 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-2] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 for region testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,719 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(1109): Closing testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf.: disabling compactions & flushes 2014-08-24 06:15:46,719 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(1136): Updates disabled for region testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,719 INFO [StoreCloserThread-testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:46,720 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:46,720 INFO [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(1218): Closed testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,720 DEBUG [FifoRpcScheduler.handler7-thread-3] master.AssignmentManager(4068): Got transition CLOSED for {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=PENDING_CLOSE, ts=1408860946714, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,721 INFO [PriorityRpcServer.handler=2,queue=0,port=51857] regionserver.HRegionServer(3859): Close 98d257181059e18e5d88f247eaefedfb, via zk=no, znode version=-1, on null 2014-08-24 06:15:46,721 DEBUG [Thread-922] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@627ad5a0 2014-08-24 06:15:46,721 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] handler.CloseRegionHandler(128): Processing close of testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. 2014-08-24 06:15:46,721 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-1] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 for region testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. 2014-08-24 06:15:46,722 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(1109): Closing testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb.: disabling compactions & flushes 2014-08-24 06:15:46,722 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(802): Transition {d3f4637d404da158b76d1a44746ca2ef state=PENDING_CLOSE, ts=1408860946713, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {d3f4637d404da158b76d1a44746ca2ef state=OFFLINE, ts=1408860946722, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:46,722 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(1136): Updates disabled for region testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. 2014-08-24 06:15:46,722 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStateStore(183): Updating row testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. with state=OFFLINE 2014-08-24 06:15:46,722 INFO [StoreCloserThread-testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:46,722 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:46,723 INFO [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(1218): Closed testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. 2014-08-24 06:15:46,723 DEBUG [Thread-922] client.HBaseAdmin(956): Sleeping= 100ms, waiting for all regions to be disabled in testMoveRegion 2014-08-24 06:15:46,723 DEBUG [FifoRpcScheduler.handler7-thread-2] master.AssignmentManager(4068): Got transition CLOSED for {98d257181059e18e5d88f247eaefedfb state=PENDING_CLOSE, ts=1408860946713, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,723 INFO [PriorityRpcServer.handler=8,queue=0,port=58042] regionserver.HRegionServer(3859): Close 28f6d3e65d1c195f5df4cef92084bd8c, via zk=no, znode version=-1, on null 2014-08-24 06:15:46,723 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] handler.CloseRegionHandler(128): Processing close of testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. 2014-08-24 06:15:46,723 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-3] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 for region testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. 2014-08-24 06:15:46,724 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(1109): Closing testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c.: disabling compactions & flushes 2014-08-24 06:15:46,724 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(1136): Updates disabled for region testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. 2014-08-24 06:15:46,724 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(802): Transition {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=PENDING_CLOSE, ts=1408860946714, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {ceafd98e56e0ec8e4fb7c47fbbc93bbf state=OFFLINE, ts=1408860946724, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:46,724 INFO [StoreCloserThread-testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:46,724 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStateStore(183): Updating row testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. with state=OFFLINE 2014-08-24 06:15:46,725 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:46,725 INFO [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] regionserver.HRegion(1218): Closed testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. 2014-08-24 06:15:46,725 DEBUG [FifoRpcScheduler.handler7-thread-5] master.AssignmentManager(4068): Got transition CLOSED for {28f6d3e65d1c195f5df4cef92084bd8c state=PENDING_CLOSE, ts=1408860946714, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,725 INFO [PriorityRpcServer.handler=9,queue=0,port=51857] regionserver.HRegionServer(3859): Close 3ffbbb0df93a5a80e386b6d2ac574e03, via zk=no, znode version=-1, on null 2014-08-24 06:15:46,726 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] handler.CloseRegionHandler(128): Processing close of testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. 2014-08-24 06:15:46,726 DEBUG [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-4] master.AssignmentManager(1825): Sent CLOSE to ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 for region testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. 2014-08-24 06:15:46,726 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(802): Transition {98d257181059e18e5d88f247eaefedfb state=PENDING_CLOSE, ts=1408860946713, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {98d257181059e18e5d88f247eaefedfb state=OFFLINE, ts=1408860946726, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:46,726 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(1109): Closing testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03.: disabling compactions & flushes 2014-08-24 06:15:46,726 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStateStore(183): Updating row testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. with state=OFFLINE 2014-08-24 06:15:46,726 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(1136): Updates disabled for region testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. 2014-08-24 06:15:46,727 INFO [StoreCloserThread-testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:46,727 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:46,727 INFO [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] regionserver.HRegion(1218): Closed testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. 2014-08-24 06:15:46,727 DEBUG [FifoRpcScheduler.handler7-thread-1] master.AssignmentManager(4068): Got transition CLOSED for {3ffbbb0df93a5a80e386b6d2ac574e03 state=PENDING_CLOSE, ts=1408860946714, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,727 INFO [FifoRpcScheduler.handler7-thread-3] master.RegionStates(455): Offlined ceafd98e56e0ec8e4fb7c47fbbc93bbf from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,728 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] handler.CloseRegionHandler(182): Closed testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. 2014-08-24 06:15:46,728 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(802): Transition {28f6d3e65d1c195f5df4cef92084bd8c state=PENDING_CLOSE, ts=1408860946714, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} to {28f6d3e65d1c195f5df4cef92084bd8c state=OFFLINE, ts=1408860946728, server=ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571} 2014-08-24 06:15:46,728 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStateStore(183): Updating row testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. with state=OFFLINE 2014-08-24 06:15:46,729 INFO [FifoRpcScheduler.handler7-thread-4] master.RegionStates(455): Offlined d3f4637d404da158b76d1a44746ca2ef from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,729 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] handler.CloseRegionHandler(182): Closed testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. 2014-08-24 06:15:46,730 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(802): Transition {3ffbbb0df93a5a80e386b6d2ac574e03 state=PENDING_CLOSE, ts=1408860946714, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} to {3ffbbb0df93a5a80e386b6d2ac574e03 state=OFFLINE, ts=1408860946730, server=ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648} 2014-08-24 06:15:46,730 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStateStore(183): Updating row testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. with state=OFFLINE 2014-08-24 06:15:46,731 INFO [FifoRpcScheduler.handler7-thread-5] master.RegionStates(455): Offlined 28f6d3e65d1c195f5df4cef92084bd8c from ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:46,731 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-0] handler.CloseRegionHandler(182): Closed testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. 2014-08-24 06:15:46,733 INFO [FifoRpcScheduler.handler7-thread-2] master.RegionStates(455): Offlined 98d257181059e18e5d88f247eaefedfb from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,733 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] handler.CloseRegionHandler(182): Closed testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. 2014-08-24 06:15:46,734 INFO [FifoRpcScheduler.handler7-thread-1] master.RegionStates(455): Offlined 3ffbbb0df93a5a80e386b6d2ac574e03 from ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,735 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-0] handler.CloseRegionHandler(182): Closed testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. 2014-08-24 06:15:46,764 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler$BulkDisabler(230): Disable waiting until done; 300000 ms remaining; [] 2014-08-24 06:15:46,765 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DisableTableHandler(190): Disabled table, testMoveRegion, is done=true 2014-08-24 06:15:46,767 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testMoveRegion/write-master:578260000000001 2014-08-24 06:15:46,824 INFO [Thread-922] zookeeper.RecoverableZooKeeper(120): Process identifier=catalogtracker-on-hconnection-0x47b642c5 connecting to ZooKeeper ensemble=localhost:56428 2014-08-24 06:15:46,824 DEBUG [Thread-922] catalog.CatalogTracker(195): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@142a24c 2014-08-24 06:15:46,825 DEBUG [Thread-922-EventThread] zookeeper.ZooKeeperWatcher(312): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2014-08-24 06:15:46,825 DEBUG [Thread-922] zookeeper.ZKUtil(428): catalogtracker-on-hconnection-0x47b642c5, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/meta-region-server 2014-08-24 06:15:46,826 DEBUG [Thread-922-EventThread] zookeeper.ZooKeeperWatcher(389): catalogtracker-on-hconnection-0x47b642c5-0x14806a73b1d0044 connected 2014-08-24 06:15:46,831 DEBUG [Thread-922] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@142a24c 2014-08-24 06:15:46,832 INFO [Thread-922] client.HBaseAdmin(971): Disabled testMoveRegion 2014-08-24 06:15:46,833 INFO [FifoRpcScheduler.handler7-thread-4] master.HMaster(1816): Client=apurtell//10.229.0.82 delete testMoveRegion 2014-08-24 06:15:46,834 DEBUG [FifoRpcScheduler.handler7-thread-4] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testMoveRegion/write-master:578260000000002 2014-08-24 06:15:46,842 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.TableEventHandler(123): Handling table operation C_M_DELETE_TABLE on table testMoveRegion 2014-08-24 06:15:46,847 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(93): Deleting regions from META 2014-08-24 06:15:46,850 INFO [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] catalog.MetaEditor(495): Deleted [{ENCODED => d3f4637d404da158b76d1a44746ca2ef, NAME => 'testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef.', STARTKEY => '', ENDKEY => 'A'}, {ENCODED => 98d257181059e18e5d88f247eaefedfb, NAME => 'testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb.', STARTKEY => 'A', ENDKEY => 'I'}, {ENCODED => ceafd98e56e0ec8e4fb7c47fbbc93bbf, NAME => 'testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf.', STARTKEY => 'I', ENDKEY => 'Q'}, {ENCODED => 28f6d3e65d1c195f5df4cef92084bd8c, NAME => 'testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c.', STARTKEY => 'Q', ENDKEY => 'Z'}, {ENCODED => 3ffbbb0df93a5a80e386b6d2ac574e03, NAME => 'testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03.', STARTKEY => 'Z', ENDKEY => ''}] 2014-08-24 06:15:46,852 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testMoveRegion,,1408860946333.d3f4637d404da158b76d1a44746ca2ef. from FS 2014-08-24 06:15:46,852 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/d3f4637d404da158b76d1a44746ca2ef 2014-08-24 06:15:46,854 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/d3f4637d404da158b76d1a44746ca2ef/FAMILY] 2014-08-24 06:15:46,857 INFO [IPC Server handler 2 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741891_1067 127.0.0.1:45654 127.0.0.1:42916 127.0.0.1:60268 2014-08-24 06:15:46,858 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/d3f4637d404da158b76d1a44746ca2ef 2014-08-24 06:15:46,858 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testMoveRegion,A,1408860946333.98d257181059e18e5d88f247eaefedfb. from FS 2014-08-24 06:15:46,859 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/98d257181059e18e5d88f247eaefedfb 2014-08-24 06:15:46,860 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/98d257181059e18e5d88f247eaefedfb/FAMILY] 2014-08-24 06:15:46,863 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741892_1068 127.0.0.1:39258 127.0.0.1:45654 127.0.0.1:42916 2014-08-24 06:15:46,863 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/98d257181059e18e5d88f247eaefedfb 2014-08-24 06:15:46,863 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testMoveRegion,I,1408860946333.ceafd98e56e0ec8e4fb7c47fbbc93bbf. from FS 2014-08-24 06:15:46,864 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/ceafd98e56e0ec8e4fb7c47fbbc93bbf 2014-08-24 06:15:46,865 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/ceafd98e56e0ec8e4fb7c47fbbc93bbf/FAMILY] 2014-08-24 06:15:46,868 INFO [IPC Server handler 1 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741893_1069 127.0.0.1:39258 127.0.0.1:60268 127.0.0.1:45654 2014-08-24 06:15:46,868 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/ceafd98e56e0ec8e4fb7c47fbbc93bbf 2014-08-24 06:15:46,868 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testMoveRegion,Q,1408860946333.28f6d3e65d1c195f5df4cef92084bd8c. from FS 2014-08-24 06:15:46,868 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/28f6d3e65d1c195f5df4cef92084bd8c 2014-08-24 06:15:46,870 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/28f6d3e65d1c195f5df4cef92084bd8c/FAMILY] 2014-08-24 06:15:46,873 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741895_1071 127.0.0.1:60268 127.0.0.1:45654 127.0.0.1:39258 2014-08-24 06:15:46,873 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/28f6d3e65d1c195f5df4cef92084bd8c 2014-08-24 06:15:46,873 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(104): Archiving region testMoveRegion,Z,1408860946333.3ffbbb0df93a5a80e386b6d2ac574e03. from FS 2014-08-24 06:15:46,873 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/3ffbbb0df93a5a80e386b6d2ac574e03 2014-08-24 06:15:46,875 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/3ffbbb0df93a5a80e386b6d2ac574e03/FAMILY] 2014-08-24 06:15:46,878 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741894_1070 127.0.0.1:42916 127.0.0.1:39258 127.0.0.1:60268 2014-08-24 06:15:46,878 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:59365/user/apurtell/hbase/.tmp/data/default/testMoveRegion/3ffbbb0df93a5a80e386b6d2ac574e03 2014-08-24 06:15:46,878 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741890_1066 127.0.0.1:60268 127.0.0.1:45654 127.0.0.1:42916 2014-08-24 06:15:46,879 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(114): Table 'testMoveRegion' archived! 2014-08-24 06:15:46,879 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(117): Removing 'testMoveRegion' descriptor. 2014-08-24 06:15:46,879 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(121): Removing 'testMoveRegion' from region states. 2014-08-24 06:15:46,879 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] handler.DeleteTableHandler(125): Marking 'testMoveRegion' as deleted. 2014-08-24 06:15:46,881 DEBUG [MASTER_TABLE_OPERATIONS-ip-10-229-0-82:57826-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testMoveRegion/write-master:578260000000002 2014-08-24 06:15:46,945 INFO [Thread-922] client.HBaseAdmin(705): Deleted testMoveRegion 2014-08-24 06:15:46,958 INFO [main] hbase.ResourceChecker(171): after: master.TestZKLessAMOnCluster#testMoveRegion Thread=491 (was 488) - Thread LEAK? -, OpenFileDescriptor=704 (was 702) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=65536 (was 65536), SystemLoadAverage=39 (was 39), ProcessCount=152 (was 152), AvailableMemoryMB=27853 (was 27860), ConnectionCount=9 (was 9) 2014-08-24 06:15:46,959 INFO [main] hbase.HBaseTestingUtility(952): Shutting down minicluster 2014-08-24 06:15:46,959 DEBUG [main] util.JVMClusterUtil(237): Shutting down HBase Cluster 2014-08-24 06:15:46,959 INFO [main] master.HMaster(2391): Cluster shutdown requested 2014-08-24 06:15:46,959 INFO [M:1;ip-10-229-0-82:57826] master.ServerManager(485): Waiting on regionserver(s) to go down ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571, ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608, ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:46,959 INFO [CatalogJanitor-ip-10-229-0-82:57826] hbase.Chore(93): CatalogJanitor-ip-10-229-0-82:57826 exiting 2014-08-24 06:15:46,960 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-ClusterStatusChore] hbase.Chore(93): ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-ClusterStatusChore exiting 2014-08-24 06:15:46,960 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-BalancerChore] hbase.Chore(93): ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100-BalancerChore exiting 2014-08-24 06:15:46,961 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2014-08-24 06:15:46,961 INFO [main] regionserver.HRegionServer(1777): STOPPED: Shutdown requested 2014-08-24 06:15:46,961 DEBUG [Thread-434-EventThread] zookeeper.ZooKeeperWatcher(312): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2014-08-24 06:15:46,960 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:58042-0x14806a73b1d0002, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2014-08-24 06:15:46,961 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZKUtil(430): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2014-08-24 06:15:46,960 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2014-08-24 06:15:46,961 INFO [main] regionserver.HRegionServer(1777): STOPPED: Shutdown requested 2014-08-24 06:15:46,962 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] zookeeper.ZKUtil(430): regionserver:58042-0x14806a73b1d0002, quorum=localhost:56428, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2014-08-24 06:15:46,961 INFO [RS:1;ip-10-229-0-82:58042] regionserver.SplitLogWorker(604): Sending interrupt to stop the worker thread 2014-08-24 06:15:46,961 DEBUG [Thread-434-EventThread] zookeeper.ZKUtil(430): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2014-08-24 06:15:46,964 INFO [SplitLogWorker-ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571] regionserver.SplitLogWorker(295): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException 2014-08-24 06:15:46,964 INFO [RS:1;ip-10-229-0-82:58042] regionserver.HRegionServer(932): Stopping infoServer 2014-08-24 06:15:46,963 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZKUtil(430): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2014-08-24 06:15:46,962 INFO [RS:2;ip-10-229-0-82:55743] regionserver.SplitLogWorker(604): Sending interrupt to stop the worker thread 2014-08-24 06:15:46,962 INFO [main] regionserver.HRegionServer(1777): STOPPED: Shutdown requested 2014-08-24 06:15:46,965 INFO [SplitLogWorker-ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608] regionserver.SplitLogWorker(295): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException 2014-08-24 06:15:46,965 INFO [SplitLogWorker-ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608] regionserver.SplitLogWorker(212): SplitLogWorker ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 exiting 2014-08-24 06:15:46,965 INFO [RS:2;ip-10-229-0-82:55743] regionserver.HRegionServer(932): Stopping infoServer 2014-08-24 06:15:46,964 INFO [SplitLogWorker-ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571] regionserver.SplitLogWorker(212): SplitLogWorker ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 exiting 2014-08-24 06:15:46,966 INFO [RS:1;ip-10-229-0-82:58042] log.Slf4jLog(67): Stopped SelectChannelConnector@0.0.0.0:0 2014-08-24 06:15:46,966 INFO [RS:3;ip-10-229-0-82:51857] regionserver.SplitLogWorker(604): Sending interrupt to stop the worker thread 2014-08-24 06:15:46,966 INFO [RS:3;ip-10-229-0-82:51857] regionserver.HRegionServer(932): Stopping infoServer 2014-08-24 06:15:46,966 INFO [SplitLogWorker-ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648] regionserver.SplitLogWorker(295): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException 2014-08-24 06:15:46,966 INFO [SplitLogWorker-ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648] regionserver.SplitLogWorker(212): SplitLogWorker ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 exiting 2014-08-24 06:15:46,967 INFO [RS:2;ip-10-229-0-82:55743] log.Slf4jLog(67): Stopped SelectChannelConnector@0.0.0.0:0 2014-08-24 06:15:46,967 INFO [RS:3;ip-10-229-0-82:51857] log.Slf4jLog(67): Stopped SelectChannelConnector@0.0.0.0:0 2014-08-24 06:15:47,066 INFO [RS:1;ip-10-229-0-82:58042] snapshot.RegionServerSnapshotManager(133): Stopping RegionServerSnapshotManager gracefully. 2014-08-24 06:15:47,067 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(274): MemStoreFlusher.1 exiting 2014-08-24 06:15:47,066 INFO [RS:1;ip-10-229-0-82:58042.nonceCleaner] hbase.Chore(93): RS:1;ip-10-229-0-82:58042.nonceCleaner exiting 2014-08-24 06:15:47,067 INFO [RS:1;ip-10-229-0-82:58042] regionserver.HRegionServer(978): stopping server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:47,066 INFO [RS:1;ip-10-229-0-82:58042.logRoller] regionserver.LogRoller(120): LogRoller exiting. 2014-08-24 06:15:47,066 INFO [RS:1;ip-10-229-0-82:58042.compactionChecker] hbase.Chore(93): RS:1;ip-10-229-0-82:58042.compactionChecker exiting 2014-08-24 06:15:47,066 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(274): MemStoreFlusher.0 exiting 2014-08-24 06:15:47,067 DEBUG [RS:1;ip-10-229-0-82:58042] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@6705cb85 2014-08-24 06:15:47,068 INFO [RS:2;ip-10-229-0-82:55743.nonceCleaner] hbase.Chore(93): RS:2;ip-10-229-0-82:55743.nonceCleaner exiting 2014-08-24 06:15:47,068 INFO [RS_OPEN_META-ip-10-229-0-82:51857-0-MetaLogRoller] regionserver.LogRoller(120): LogRoller exiting. 2014-08-24 06:15:47,067 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] handler.CloseRegionHandler(128): Processing close of testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. 2014-08-24 06:15:47,067 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] handler.CloseRegionHandler(128): Processing close of testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f. 2014-08-24 06:15:47,069 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(1109): Closing testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d.: disabling compactions & flushes 2014-08-24 06:15:47,069 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(1109): Closing testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f.: disabling compactions & flushes 2014-08-24 06:15:47,068 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(274): MemStoreFlusher.0 exiting 2014-08-24 06:15:47,068 INFO [RS:3;ip-10-229-0-82:51857.nonceCleaner] hbase.Chore(93): RS:3;ip-10-229-0-82:51857.nonceCleaner exiting 2014-08-24 06:15:47,068 INFO [RS:3;ip-10-229-0-82:51857.compactionChecker] hbase.Chore(93): RS:3;ip-10-229-0-82:51857.compactionChecker exiting 2014-08-24 06:15:47,068 INFO [RS:3;ip-10-229-0-82:51857.logRoller] regionserver.LogRoller(120): LogRoller exiting. 2014-08-24 06:15:47,068 INFO [RS:3;ip-10-229-0-82:51857] snapshot.RegionServerSnapshotManager(133): Stopping RegionServerSnapshotManager gracefully. 2014-08-24 06:15:47,068 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(274): MemStoreFlusher.1 exiting 2014-08-24 06:15:47,070 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] handler.CloseRegionHandler(128): Processing close of testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3. 2014-08-24 06:15:47,068 INFO [RS:1;ip-10-229-0-82:58042] client.HConnectionManager$HConnectionImplementation(1850): Closing zookeeper sessionid=0x14806a73b1d0006 2014-08-24 06:15:47,068 INFO [RS:2;ip-10-229-0-82:55743.compactionChecker] hbase.Chore(93): RS:2;ip-10-229-0-82:55743.compactionChecker exiting 2014-08-24 06:15:47,068 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(274): MemStoreFlusher.1 exiting 2014-08-24 06:15:47,068 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(274): MemStoreFlusher.0 exiting 2014-08-24 06:15:47,068 INFO [RS:2;ip-10-229-0-82:55743.logRoller] regionserver.LogRoller(120): LogRoller exiting. 2014-08-24 06:15:47,068 INFO [RS:2;ip-10-229-0-82:55743] snapshot.RegionServerSnapshotManager(133): Stopping RegionServerSnapshotManager gracefully. 2014-08-24 06:15:47,071 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(1109): Closing testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3.: disabling compactions & flushes 2014-08-24 06:15:47,072 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-2] handler.CloseRegionHandler(128): Processing close of testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. 2014-08-24 06:15:47,072 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] handler.CloseRegionHandler(128): Processing close of testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. 2014-08-24 06:15:47,070 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] handler.CloseRegionHandler(128): Processing close of hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79. 2014-08-24 06:15:47,070 INFO [RS:3;ip-10-229-0-82:51857] regionserver.HRegionServer(978): stopping server ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:47,069 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(1136): Updates disabled for region testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f. 2014-08-24 06:15:47,069 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(1136): Updates disabled for region testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. 2014-08-24 06:15:47,073 INFO [RS:1;ip-10-229-0-82:58042] regionserver.HRegionServer(1163): Waiting on 2 regions to close 2014-08-24 06:15:47,073 DEBUG [RS:1;ip-10-229-0-82:58042] regionserver.HRegionServer(1167): {578ca2c80e167109c1bb4231a65aec1f=testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f., 8229a7b7ed833545f9d1bcb8fab66f4d=testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d.} 2014-08-24 06:15:47,073 INFO [StoreCloserThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:47,073 DEBUG [RS:3;ip-10-229-0-82:51857] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@af6d066 2014-08-24 06:15:47,073 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(1109): Closing hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79.: disabling compactions & flushes 2014-08-24 06:15:47,072 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(1109): Closing testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5.: disabling compactions & flushes 2014-08-24 06:15:47,072 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(1109): Closing testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603.: disabling compactions & flushes 2014-08-24 06:15:47,072 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] handler.CloseRegionHandler(128): Processing close of testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. 2014-08-24 06:15:47,072 INFO [RS:2;ip-10-229-0-82:55743] regionserver.HRegionServer(978): stopping server ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:47,072 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(1136): Updates disabled for region testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3. 2014-08-24 06:15:47,074 DEBUG [RS:2;ip-10-229-0-82:55743] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@41a95a1a 2014-08-24 06:15:47,074 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(1109): Closing testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb.: disabling compactions & flushes 2014-08-24 06:15:47,074 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(1136): Updates disabled for region testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. 2014-08-24 06:15:47,074 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(1136): Updates disabled for region testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. 2014-08-24 06:15:47,073 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(1136): Updates disabled for region hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79. 2014-08-24 06:15:47,073 INFO [RS:3;ip-10-229-0-82:51857] client.HConnectionManager$HConnectionImplementation(1850): Closing zookeeper sessionid=0x14806a73b1d0007 2014-08-24 06:15:47,075 INFO [StoreCloserThread-testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:47,073 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:47,073 INFO [StoreCloserThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:47,075 INFO [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] regionserver.HRegion(1218): Closed testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f. 2014-08-24 06:15:47,075 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:47,074 INFO [StoreCloserThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:47,074 INFO [StoreCloserThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:47,074 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(1136): Updates disabled for region testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. 2014-08-24 06:15:47,076 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:47,076 INFO [StoreCloserThread-hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79.-1] regionserver.HStore(759): Closed info 2014-08-24 06:15:47,075 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-2] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:47,075 INFO [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] regionserver.HRegion(1218): Closed testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. 2014-08-24 06:15:47,075 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-2] handler.CloseRegionHandler(182): Closed testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Q,1408860944434.578ca2c80e167109c1bb4231a65aec1f. 2014-08-24 06:15:47,075 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:47,077 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-1] handler.CloseRegionHandler(182): Closed testOfflineRegion,I,1408860944921.5fa8d7f89143dac7fd49a75d32df5fe5. 2014-08-24 06:15:47,077 INFO [RS_CLOSE_REGION-ip-10-229-0-82:55743-2] regionserver.HRegion(1218): Closed testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. 2014-08-24 06:15:47,077 INFO [StoreCloserThread-testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb.-1] regionserver.HStore(759): Closed FAMILY 2014-08-24 06:15:47,076 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:47,076 INFO [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] regionserver.HRegion(1218): Closed testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3. 2014-08-24 06:15:47,076 INFO [RS:3;ip-10-229-0-82:51857] regionserver.CompactSplitThread(360): Waiting for Split Thread to finish... 2014-08-24 06:15:47,076 INFO [RS:2;ip-10-229-0-82:55743] client.HConnectionManager$HConnectionImplementation(1850): Closing zookeeper sessionid=0x14806a73b1d0009 2014-08-24 06:15:47,078 INFO [RS:3;ip-10-229-0-82:51857] regionserver.CompactSplitThread(360): Waiting for Merge Thread to finish... 2014-08-24 06:15:47,077 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-1] handler.CloseRegionHandler(182): Closed testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,A,1408860944434.e859174d1aec5977deb209db04be31d3. 2014-08-24 06:15:47,077 INFO [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] regionserver.HRegion(1218): Closed hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79. 2014-08-24 06:15:47,077 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:47,077 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-2] handler.CloseRegionHandler(182): Closed testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,Z,1408860944435.39b8af357f05404077613e6bb6e5c603. 2014-08-24 06:15:47,077 INFO [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] regionserver.HRegion(1218): Closed testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. 2014-08-24 06:15:47,078 INFO [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] regionserver.HRegion(1218): Closed testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. 2014-08-24 06:15:47,078 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:51857-2] handler.CloseRegionHandler(182): Closed hbase:namespace,,1408860898339.43a5af54b31ca15a1e49e35d4ace9f79. 2014-08-24 06:15:47,078 INFO [RS:3;ip-10-229-0-82:51857] regionserver.CompactSplitThread(360): Waiting for Large Compaction Thread to finish... 2014-08-24 06:15:47,078 INFO [RS:3;ip-10-229-0-82:51857] regionserver.CompactSplitThread(360): Waiting for Small Compaction Thread to finish... 2014-08-24 06:15:47,078 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:55743-0] handler.CloseRegionHandler(182): Closed testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,,1408860944434.4d4fec75cd98afbba2cf4df10292c3eb. 2014-08-24 06:15:47,078 DEBUG [RS_CLOSE_REGION-ip-10-229-0-82:58042-1] handler.CloseRegionHandler(182): Closed testSSHWhenDisablingTableRegionsInOpeningOrPendingOpenState,I,1408860944434.8229a7b7ed833545f9d1bcb8fab66f4d. 2014-08-24 06:15:47,079 INFO [RS:2;ip-10-229-0-82:55743] regionserver.HRegionServer(997): stopping server ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608; all regions closed. 2014-08-24 06:15:47,079 INFO [RS:3;ip-10-229-0-82:51857] regionserver.HRegionServer(1163): Waiting on 1 regions to close 2014-08-24 06:15:47,079 DEBUG [RS:3;ip-10-229-0-82:51857] regionserver.HRegionServer(1167): {1588230740=hbase:meta,,1.1588230740} 2014-08-24 06:15:47,079 DEBUG [RS:2;ip-10-229-0-82:55743-WAL.AsyncNotifier] wal.FSHLog$AsyncNotifier(1332): RS:2;ip-10-229-0-82:55743-WAL.AsyncNotifier interrupted while waiting for notification from AsyncSyncer thread 2014-08-24 06:15:47,079 INFO [RS:2;ip-10-229-0-82:55743-WAL.AsyncNotifier] wal.FSHLog$AsyncNotifier(1337): RS:2;ip-10-229-0-82:55743-WAL.AsyncNotifier exiting 2014-08-24 06:15:47,079 DEBUG [RS_CLOSE_META-ip-10-229-0-82:51857-0] handler.CloseRegionHandler(128): Processing close of hbase:meta,,1.1588230740 2014-08-24 06:15:47,079 DEBUG [RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer0] wal.FSHLog$AsyncSyncer(1278): RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer0 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,079 INFO [RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer0] wal.FSHLog$AsyncSyncer(1283): RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer0 exiting 2014-08-24 06:15:47,079 DEBUG [RS_CLOSE_META-ip-10-229-0-82:51857-0] regionserver.HRegion(1109): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2014-08-24 06:15:47,079 DEBUG [RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer1] wal.FSHLog$AsyncSyncer(1278): RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer1 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,080 INFO [RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer1] wal.FSHLog$AsyncSyncer(1283): RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer1 exiting 2014-08-24 06:15:47,080 DEBUG [RS_CLOSE_META-ip-10-229-0-82:51857-0] regionserver.HRegion(1136): Updates disabled for region hbase:meta,,1.1588230740 2014-08-24 06:15:47,080 DEBUG [RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer2] wal.FSHLog$AsyncSyncer(1278): RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer2 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,080 INFO [RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer2] wal.FSHLog$AsyncSyncer(1283): RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer2 exiting 2014-08-24 06:15:47,080 INFO [RS_CLOSE_META-ip-10-229-0-82:51857-0] regionserver.HRegion(1689): Started memstore flush for hbase:meta,,1.1588230740, current region memstore size 141.4 K 2014-08-24 06:15:47,080 DEBUG [RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer3] wal.FSHLog$AsyncSyncer(1278): RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer3 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,080 INFO [RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer3] wal.FSHLog$AsyncSyncer(1283): RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer3 exiting 2014-08-24 06:15:47,080 DEBUG [RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer4] wal.FSHLog$AsyncSyncer(1278): RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer4 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,080 INFO [RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer4] wal.FSHLog$AsyncSyncer(1283): RS:2;ip-10-229-0-82:55743-WAL.AsyncSyncer4 exiting 2014-08-24 06:15:47,081 DEBUG [RS:2;ip-10-229-0-82:55743-WAL.AsyncWriter] wal.FSHLog$AsyncWriter(1153): RS:2;ip-10-229-0-82:55743-WAL.AsyncWriter interrupted while waiting for newer writes added to local buffer 2014-08-24 06:15:47,081 INFO [RS:2;ip-10-229-0-82:55743-WAL.AsyncWriter] wal.FSHLog$AsyncWriter(1158): RS:2;ip-10-229-0-82:55743-WAL.AsyncWriter exiting 2014-08-24 06:15:47,081 DEBUG [RS:2;ip-10-229-0-82:55743] wal.FSHLog(938): Closing WAL writer in hdfs://localhost:59365/user/apurtell/hbase/WALs/ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:47,084 INFO [IPC Server handler 7 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741831_1007{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:47,084 INFO [IPC Server handler 8 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741831_1007{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW]]} size 0 2014-08-24 06:15:47,085 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741831_1007 size 91 2014-08-24 06:15:47,087 DEBUG [RS:2;ip-10-229-0-82:55743] wal.FSHLog(882): Moved 1 WAL file(s) to /user/apurtell/hbase/oldWALs 2014-08-24 06:15:47,094 INFO [IPC Server handler 6 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741896_1072{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:47,094 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741896_1072{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:42916|RBW], ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW]]} size 0 2014-08-24 06:15:47,095 INFO [IPC Server handler 0 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741896_1072 size 7104 2014-08-24 06:15:47,095 INFO [RS_CLOSE_META-ip-10-229-0-82:51857-0] regionserver.DefaultStoreFlusher(82): Flushed, sequenceid=292, memsize=12.9 K, hasBloomFilter=false, into tmp file hdfs://localhost:59365/user/apurtell/hbase/data/hbase/meta/1588230740/.tmp/0321970d77b040f4b394f3ec269f2d87 2014-08-24 06:15:47,101 INFO [RS_CLOSE_META-ip-10-229-0-82:51857-0] regionserver.StoreFile$Reader(1382): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0321970d77b040f4b394f3ec269f2d87 2014-08-24 06:15:47,102 DEBUG [RS_CLOSE_META-ip-10-229-0-82:51857-0] regionserver.HRegionFileSystem(376): Committing store file hdfs://localhost:59365/user/apurtell/hbase/data/hbase/meta/1588230740/.tmp/0321970d77b040f4b394f3ec269f2d87 as hdfs://localhost:59365/user/apurtell/hbase/data/hbase/meta/1588230740/info/0321970d77b040f4b394f3ec269f2d87 2014-08-24 06:15:47,107 INFO [RS_CLOSE_META-ip-10-229-0-82:51857-0] regionserver.StoreFile$Reader(1382): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0321970d77b040f4b394f3ec269f2d87 2014-08-24 06:15:47,107 INFO [RS_CLOSE_META-ip-10-229-0-82:51857-0] regionserver.HStore(862): Added hdfs://localhost:59365/user/apurtell/hbase/data/hbase/meta/1588230740/info/0321970d77b040f4b394f3ec269f2d87, entries=56, sequenceid=292, filesize=6.9 K 2014-08-24 06:15:47,107 INFO [RS_CLOSE_META-ip-10-229-0-82:51857-0] regionserver.HRegion(1835): Finished memstore flush of ~141.4 K/144816, currentsize=0/0 for region hbase:meta,,1.1588230740 in 27ms, sequenceid=292, compaction requested=false 2014-08-24 06:15:47,109 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(759): Closed info 2014-08-24 06:15:47,109 DEBUG [RS_CLOSE_META-ip-10-229-0-82:51857-0] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster$MyRegionObserver 2014-08-24 06:15:47,109 DEBUG [RS_CLOSE_META-ip-10-229-0-82:51857-0] coprocessor.CoprocessorHost(282): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2014-08-24 06:15:47,109 INFO [RS_CLOSE_META-ip-10-229-0-82:51857-0] regionserver.HRegion(1218): Closed hbase:meta,,1.1588230740 2014-08-24 06:15:47,109 DEBUG [RS_CLOSE_META-ip-10-229-0-82:51857-0] handler.CloseRegionHandler(182): Closed hbase:meta,,1.1588230740 2014-08-24 06:15:47,155 INFO [ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100.splitLogManagerTimeoutMonitor] hbase.Chore(93): ip-10-229-0-82.us-west-2.compute.internal,57826,1408860900100.splitLogManagerTimeoutMonitor exiting 2014-08-24 06:15:47,188 INFO [RS:2;ip-10-229-0-82:55743] regionserver.Leases(147): RS:2;ip-10-229-0-82:55743 closing leases 2014-08-24 06:15:47,188 INFO [RS:2;ip-10-229-0-82:55743] regionserver.Leases(150): RS:2;ip-10-229-0-82:55743 closed leases 2014-08-24 06:15:47,221 INFO [RS:2;ip-10-229-0-82:55743.periodicFlusher] hbase.Chore(93): RS:2;ip-10-229-0-82:55743.periodicFlusher exiting 2014-08-24 06:15:47,221 INFO [RS:2;ip-10-229-0-82:55743] regionserver.CompactSplitThread(360): Waiting for Split Thread to finish... 2014-08-24 06:15:47,222 INFO [RS:2;ip-10-229-0-82:55743] regionserver.CompactSplitThread(360): Waiting for Merge Thread to finish... 2014-08-24 06:15:47,222 INFO [RS:2;ip-10-229-0-82:55743] regionserver.CompactSplitThread(360): Waiting for Large Compaction Thread to finish... 2014-08-24 06:15:47,222 INFO [RS:2;ip-10-229-0-82:55743] regionserver.CompactSplitThread(360): Waiting for Small Compaction Thread to finish... 2014-08-24 06:15:47,222 INFO [RS:3;ip-10-229-0-82:51857.periodicFlusher] hbase.Chore(93): RS:3;ip-10-229-0-82:51857.periodicFlusher exiting 2014-08-24 06:15:47,222 INFO [RS:3;ip-10-229-0-82:51857.leaseChecker] regionserver.Leases(147): RS:3;ip-10-229-0-82:51857.leaseChecker closing leases 2014-08-24 06:15:47,222 INFO [RS:2;ip-10-229-0-82:55743.leaseChecker] regionserver.Leases(147): RS:2;ip-10-229-0-82:55743.leaseChecker closing leases 2014-08-24 06:15:47,223 INFO [RS:3;ip-10-229-0-82:51857.leaseChecker] regionserver.Leases(150): RS:3;ip-10-229-0-82:51857.leaseChecker closed leases 2014-08-24 06:15:47,223 INFO [RS:2;ip-10-229-0-82:55743.leaseChecker] regionserver.Leases(150): RS:2;ip-10-229-0-82:55743.leaseChecker closed leases 2014-08-24 06:15:47,223 INFO [RS:2;ip-10-229-0-82:55743] client.HConnectionManager$HConnectionImplementation(1850): Closing zookeeper sessionid=0x14806a73b1d000c 2014-08-24 06:15:47,224 INFO [RS:1;ip-10-229-0-82:58042.periodicFlusher] hbase.Chore(93): RS:1;ip-10-229-0-82:58042.periodicFlusher exiting 2014-08-24 06:15:47,224 INFO [RS:1;ip-10-229-0-82:58042.leaseChecker] regionserver.Leases(147): RS:1;ip-10-229-0-82:58042.leaseChecker closing leases 2014-08-24 06:15:47,224 INFO [RS:1;ip-10-229-0-82:58042.leaseChecker] regionserver.Leases(150): RS:1;ip-10-229-0-82:58042.leaseChecker closed leases 2014-08-24 06:15:47,226 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:58042-0x14806a73b1d0002, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:47,226 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:47,226 DEBUG [Thread-434-EventThread] zookeeper.ZooKeeperWatcher(312): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:47,226 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 2014-08-24 06:15:47,226 INFO [Thread-434-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608] 2014-08-24 06:15:47,226 DEBUG [RS:2;ip-10-229-0-82:55743-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:55743-0x14806a73b1d0004, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2014-08-24 06:15:47,226 INFO [Thread-434-EventThread] master.ServerManager(531): Cluster shutdown set; ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608 expired; onlineServers=2 2014-08-24 06:15:47,226 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:58042-0x14806a73b1d0002, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2014-08-24 06:15:47,226 DEBUG [Thread-434-EventThread] zookeeper.ZooKeeperWatcher(312): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2014-08-24 06:15:47,226 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2014-08-24 06:15:47,227 INFO [RS:2;ip-10-229-0-82:55743] regionserver.HRegionServer(1035): stopping server ip-10-229-0-82.us-west-2.compute.internal,55743,1408860896608; zookeeper connection closed. 2014-08-24 06:15:47,227 INFO [RS:2;ip-10-229-0-82:55743] regionserver.HRegionServer(1038): RS:2;ip-10-229-0-82:55743 exiting 2014-08-24 06:15:47,227 DEBUG [Thread-434-EventThread] zookeeper.ZKUtil(428): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:47,227 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@71e0da00] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(190): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@71e0da00 2014-08-24 06:15:47,227 DEBUG [Thread-434-EventThread] zookeeper.ZKUtil(428): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:47,228 DEBUG [Thread-434-EventThread] zookeeper.RegionServerTracker(91): RS node: /hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 data: PBUF�� 2014-08-24 06:15:47,228 DEBUG [Thread-434-EventThread] zookeeper.RegionServerTracker(91): RS node: /hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 data: PBUF�� 2014-08-24 06:15:47,273 INFO [RS:1;ip-10-229-0-82:58042] regionserver.HRegionServer(997): stopping server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571; all regions closed. 2014-08-24 06:15:47,273 DEBUG [RS:1;ip-10-229-0-82:58042-WAL.AsyncNotifier] wal.FSHLog$AsyncNotifier(1332): RS:1;ip-10-229-0-82:58042-WAL.AsyncNotifier interrupted while waiting for notification from AsyncSyncer thread 2014-08-24 06:15:47,273 INFO [RS:1;ip-10-229-0-82:58042-WAL.AsyncNotifier] wal.FSHLog$AsyncNotifier(1337): RS:1;ip-10-229-0-82:58042-WAL.AsyncNotifier exiting 2014-08-24 06:15:47,274 DEBUG [RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer0] wal.FSHLog$AsyncSyncer(1278): RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer0 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,274 INFO [RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer0] wal.FSHLog$AsyncSyncer(1283): RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer0 exiting 2014-08-24 06:15:47,274 DEBUG [RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer1] wal.FSHLog$AsyncSyncer(1278): RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer1 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,274 INFO [RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer1] wal.FSHLog$AsyncSyncer(1283): RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer1 exiting 2014-08-24 06:15:47,274 DEBUG [RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer2] wal.FSHLog$AsyncSyncer(1278): RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer2 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,274 INFO [RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer2] wal.FSHLog$AsyncSyncer(1283): RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer2 exiting 2014-08-24 06:15:47,274 DEBUG [RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer3] wal.FSHLog$AsyncSyncer(1278): RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer3 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,274 INFO [RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer3] wal.FSHLog$AsyncSyncer(1283): RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer3 exiting 2014-08-24 06:15:47,275 DEBUG [RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer4] wal.FSHLog$AsyncSyncer(1278): RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer4 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,275 INFO [RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer4] wal.FSHLog$AsyncSyncer(1283): RS:1;ip-10-229-0-82:58042-WAL.AsyncSyncer4 exiting 2014-08-24 06:15:47,275 DEBUG [RS:1;ip-10-229-0-82:58042-WAL.AsyncWriter] wal.FSHLog$AsyncWriter(1153): RS:1;ip-10-229-0-82:58042-WAL.AsyncWriter interrupted while waiting for newer writes added to local buffer 2014-08-24 06:15:47,275 INFO [RS:1;ip-10-229-0-82:58042-WAL.AsyncWriter] wal.FSHLog$AsyncWriter(1158): RS:1;ip-10-229-0-82:58042-WAL.AsyncWriter exiting 2014-08-24 06:15:47,275 DEBUG [RS:1;ip-10-229-0-82:58042] wal.FSHLog(938): Closing WAL writer in hdfs://localhost:59365/user/apurtell/hbase/WALs/ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:47,278 INFO [IPC Server handler 9 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741830_1006{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:47,278 INFO [IPC Server handler 7 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741830_1006{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:47,279 INFO [IPC Server handler 8 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741830_1006 size 91 2014-08-24 06:15:47,279 INFO [RS:3;ip-10-229-0-82:51857] regionserver.HRegionServer(997): stopping server ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648; all regions closed. 2014-08-24 06:15:47,279 DEBUG [RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncNotifier] wal.FSHLog$AsyncNotifier(1332): RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncNotifier interrupted while waiting for notification from AsyncSyncer thread 2014-08-24 06:15:47,279 INFO [RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncNotifier] wal.FSHLog$AsyncNotifier(1337): RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncNotifier exiting 2014-08-24 06:15:47,279 DEBUG [RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer0] wal.FSHLog$AsyncSyncer(1278): RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer0 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,280 INFO [RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer0] wal.FSHLog$AsyncSyncer(1283): RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer0 exiting 2014-08-24 06:15:47,280 DEBUG [RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer1] wal.FSHLog$AsyncSyncer(1278): RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer1 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,280 INFO [RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer1] wal.FSHLog$AsyncSyncer(1283): RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer1 exiting 2014-08-24 06:15:47,280 DEBUG [RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer2] wal.FSHLog$AsyncSyncer(1278): RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer2 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,280 INFO [RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer2] wal.FSHLog$AsyncSyncer(1283): RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer2 exiting 2014-08-24 06:15:47,280 DEBUG [RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer3] wal.FSHLog$AsyncSyncer(1278): RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer3 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,280 INFO [RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer3] wal.FSHLog$AsyncSyncer(1283): RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer3 exiting 2014-08-24 06:15:47,281 DEBUG [RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer4] wal.FSHLog$AsyncSyncer(1278): RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer4 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,281 INFO [RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer4] wal.FSHLog$AsyncSyncer(1283): RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncSyncer4 exiting 2014-08-24 06:15:47,281 DEBUG [RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncWriter] wal.FSHLog$AsyncWriter(1153): RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncWriter interrupted while waiting for newer writes added to local buffer 2014-08-24 06:15:47,281 INFO [RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncWriter] wal.FSHLog$AsyncWriter(1158): RS_OPEN_META-ip-10-229-0-82:51857-0-WAL.AsyncWriter exiting 2014-08-24 06:15:47,281 DEBUG [RS:3;ip-10-229-0-82:51857] wal.FSHLog(938): Closing WAL writer in hdfs://localhost:59365/user/apurtell/hbase/WALs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:47,281 DEBUG [RS:1;ip-10-229-0-82:58042] wal.FSHLog(882): Moved 1 WAL file(s) to /user/apurtell/hbase/oldWALs 2014-08-24 06:15:47,283 INFO [IPC Server handler 2 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741834_1010{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:47,284 INFO [IPC Server handler 9 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45654 is added to blk_1073741834_1010{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:45654|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:47,284 INFO [IPC Server handler 5 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741834_1010 size 84166 2014-08-24 06:15:47,284 DEBUG [RS:3;ip-10-229-0-82:51857-WAL.AsyncNotifier] wal.FSHLog$AsyncNotifier(1332): RS:3;ip-10-229-0-82:51857-WAL.AsyncNotifier interrupted while waiting for notification from AsyncSyncer thread 2014-08-24 06:15:47,284 INFO [RS:3;ip-10-229-0-82:51857-WAL.AsyncNotifier] wal.FSHLog$AsyncNotifier(1337): RS:3;ip-10-229-0-82:51857-WAL.AsyncNotifier exiting 2014-08-24 06:15:47,284 DEBUG [RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer0] wal.FSHLog$AsyncSyncer(1278): RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer0 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,285 INFO [RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer0] wal.FSHLog$AsyncSyncer(1283): RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer0 exiting 2014-08-24 06:15:47,285 DEBUG [RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer1] wal.FSHLog$AsyncSyncer(1278): RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer1 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,285 INFO [RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer1] wal.FSHLog$AsyncSyncer(1283): RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer1 exiting 2014-08-24 06:15:47,285 DEBUG [RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer2] wal.FSHLog$AsyncSyncer(1278): RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer2 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,285 INFO [RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer2] wal.FSHLog$AsyncSyncer(1283): RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer2 exiting 2014-08-24 06:15:47,285 DEBUG [RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer3] wal.FSHLog$AsyncSyncer(1278): RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer3 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,285 INFO [RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer3] wal.FSHLog$AsyncSyncer(1283): RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer3 exiting 2014-08-24 06:15:47,285 DEBUG [RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer4] wal.FSHLog$AsyncSyncer(1278): RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer4 interrupted while waiting for notification from AsyncWriter thread 2014-08-24 06:15:47,285 INFO [RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer4] wal.FSHLog$AsyncSyncer(1283): RS:3;ip-10-229-0-82:51857-WAL.AsyncSyncer4 exiting 2014-08-24 06:15:47,286 DEBUG [RS:3;ip-10-229-0-82:51857-WAL.AsyncWriter] wal.FSHLog$AsyncWriter(1153): RS:3;ip-10-229-0-82:51857-WAL.AsyncWriter interrupted while waiting for newer writes added to local buffer 2014-08-24 06:15:47,286 INFO [RS:3;ip-10-229-0-82:51857-WAL.AsyncWriter] wal.FSHLog$AsyncWriter(1158): RS:3;ip-10-229-0-82:51857-WAL.AsyncWriter exiting 2014-08-24 06:15:47,286 DEBUG [RS:3;ip-10-229-0-82:51857] wal.FSHLog(938): Closing WAL writer in hdfs://localhost:59365/user/apurtell/hbase/WALs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:47,288 INFO [IPC Server handler 8 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42916 is added to blk_1073741833_1009{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:47,289 INFO [IPC Server handler 3 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60268 is added to blk_1073741833_1009{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[127.0.0.1:39258|RBW], ReplicaUnderConstruction[127.0.0.1:60268|RBW], ReplicaUnderConstruction[127.0.0.1:42916|RBW]]} size 0 2014-08-24 06:15:47,289 INFO [IPC Server handler 4 on 59365] blockmanagement.BlockManager(2237): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39258 is added to blk_1073741833_1009 size 91 2014-08-24 06:15:47,293 DEBUG [RS:3;ip-10-229-0-82:51857] wal.FSHLog(882): Moved 2 WAL file(s) to /user/apurtell/hbase/oldWALs 2014-08-24 06:15:47,382 INFO [RS:1;ip-10-229-0-82:58042] regionserver.Leases(147): RS:1;ip-10-229-0-82:58042 closing leases 2014-08-24 06:15:47,382 INFO [RS:1;ip-10-229-0-82:58042] regionserver.Leases(150): RS:1;ip-10-229-0-82:58042 closed leases 2014-08-24 06:15:47,383 INFO [RS:1;ip-10-229-0-82:58042] regionserver.CompactSplitThread(360): Waiting for Split Thread to finish... 2014-08-24 06:15:47,383 INFO [RS:1;ip-10-229-0-82:58042] regionserver.CompactSplitThread(360): Waiting for Merge Thread to finish... 2014-08-24 06:15:47,383 INFO [RS:1;ip-10-229-0-82:58042] regionserver.CompactSplitThread(360): Waiting for Large Compaction Thread to finish... 2014-08-24 06:15:47,383 INFO [RS:1;ip-10-229-0-82:58042] regionserver.CompactSplitThread(360): Waiting for Small Compaction Thread to finish... 2014-08-24 06:15:47,384 INFO [RS:1;ip-10-229-0-82:58042] client.HConnectionManager$HConnectionImplementation(1850): Closing zookeeper sessionid=0x14806a73b1d000e 2014-08-24 06:15:47,387 DEBUG [RS:1;ip-10-229-0-82:58042-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:58042-0x14806a73b1d0002, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:47,387 DEBUG [Thread-434-EventThread] zookeeper.ZooKeeperWatcher(312): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:47,387 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 2014-08-24 06:15:47,387 INFO [Thread-434-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571] 2014-08-24 06:15:47,387 INFO [Thread-434-EventThread] master.ServerManager(531): Cluster shutdown set; ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571 expired; onlineServers=1 2014-08-24 06:15:47,387 DEBUG [Thread-434-EventThread] zookeeper.ZooKeeperWatcher(312): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2014-08-24 06:15:47,388 INFO [RS:1;ip-10-229-0-82:58042] regionserver.HRegionServer(1035): stopping server ip-10-229-0-82.us-west-2.compute.internal,58042,1408860896571; zookeeper connection closed. 2014-08-24 06:15:47,388 INFO [RS:1;ip-10-229-0-82:58042] regionserver.HRegionServer(1038): RS:1;ip-10-229-0-82:58042 exiting 2014-08-24 06:15:47,388 DEBUG [Thread-434-EventThread] zookeeper.ZKUtil(428): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:47,388 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2e2c3582] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(190): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2e2c3582 2014-08-24 06:15:47,389 DEBUG [Thread-434-EventThread] zookeeper.RegionServerTracker(91): RS node: /hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 data: PBUF�� 2014-08-24 06:15:47,393 INFO [RS:3;ip-10-229-0-82:51857] regionserver.Leases(147): RS:3;ip-10-229-0-82:51857 closing leases 2014-08-24 06:15:47,394 INFO [RS:3;ip-10-229-0-82:51857] regionserver.Leases(150): RS:3;ip-10-229-0-82:51857 closed leases 2014-08-24 06:15:47,395 INFO [RS:3;ip-10-229-0-82:51857] client.HConnectionManager$HConnectionImplementation(1850): Closing zookeeper sessionid=0x14806a73b1d000b 2014-08-24 06:15:47,397 DEBUG [RS:3;ip-10-229-0-82:51857-EventThread] zookeeper.ZooKeeperWatcher(312): regionserver:51857-0x14806a73b1d0003, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:47,397 DEBUG [Thread-434-EventThread] zookeeper.ZooKeeperWatcher(312): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 2014-08-24 06:15:47,397 INFO [Thread-434-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648] 2014-08-24 06:15:47,398 INFO [Thread-434-EventThread] master.ServerManager(531): Cluster shutdown set; ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648 expired; onlineServers=0 2014-08-24 06:15:47,398 DEBUG [M:1;ip-10-229-0-82:57826] master.HMaster(1226): Stopping service threads 2014-08-24 06:15:47,398 INFO [Thread-434-EventThread] master.HMaster(2391): Cluster shutdown set; onlineServer=0 2014-08-24 06:15:47,398 DEBUG [Thread-434-EventThread] catalog.CatalogTracker(211): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@34c93bf8 2014-08-24 06:15:47,398 DEBUG [Thread-434-EventThread] zookeeper.ZooKeeperWatcher(312): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2014-08-24 06:15:47,398 INFO [RS:3;ip-10-229-0-82:51857] regionserver.HRegionServer(1035): stopping server ip-10-229-0-82.us-west-2.compute.internal,51857,1408860896648; zookeeper connection closed. 2014-08-24 06:15:47,399 INFO [M:1;ip-10-229-0-82:57826.archivedHFileCleaner] hbase.Chore(93): M:1;ip-10-229-0-82:57826.archivedHFileCleaner exiting 2014-08-24 06:15:47,399 INFO [RS:3;ip-10-229-0-82:51857] regionserver.HRegionServer(1038): RS:3;ip-10-229-0-82:51857 exiting 2014-08-24 06:15:47,399 INFO [M:1;ip-10-229-0-82:57826.oldLogCleaner] hbase.Chore(93): M:1;ip-10-229-0-82:57826.oldLogCleaner exiting 2014-08-24 06:15:47,400 INFO [M:1;ip-10-229-0-82:57826.oldLogCleaner] master.ReplicationLogCleaner(137): Stopping replicationLogCleaner-0x14806a73b1d001d, quorum=localhost:56428, baseZNode=/hbase 2014-08-24 06:15:47,400 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5d675d32] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(190): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5d675d32 2014-08-24 06:15:47,401 INFO [main] util.JVMClusterUtil(313): Shutdown of 2 master(s) and 3 regionserver(s) complete 2014-08-24 06:15:47,401 INFO [main] client.HConnectionManager$HConnectionImplementation(1850): Closing zookeeper sessionid=0x14806a73b1d0012 2014-08-24 06:15:47,404 DEBUG [Thread-434-EventThread] zookeeper.ZooKeeperWatcher(312): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2014-08-24 06:15:47,405 INFO [main] client.HConnectionManager$HConnectionImplementation(1850): Closing zookeeper sessionid=0x14806a73b1d0011 2014-08-24 06:15:47,405 DEBUG [Thread-434-EventThread] zookeeper.ZKUtil(430): master:57826-0x14806a73b1d001b, quorum=localhost:56428, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2014-08-24 06:15:47,406 INFO [main] client.HConnectionManager$HConnectionImplementation(1850): Closing zookeeper sessionid=0x14806a73b1d0010 2014-08-24 06:15:47,407 INFO [main] client.HConnectionManager$HConnectionImplementation(1850): Closing zookeeper sessionid=0x14806a73b1d000f 2014-08-24 06:15:47,409 ERROR [M:1;ip-10-229-0-82:57826.oldLogCleaner] client.HConnectionManager(488): Connection not found in the list, can't delete it (connection key=HConnectionKey{properties={hbase.rpc.timeout=60000, hbase.zookeeper.property.clientPort=56428, hbase.client.pause=100, zookeeper.znode.parent=/hbase, hbase.client.retries.number=350, hbase.zookeeper.quorum=localhost}, username='apurtell.hfs.4'}). May be the key was modified? java.lang.Exception at org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:488) at org.apache.hadoop.hbase.client.HConnectionManager.deleteConnection(HConnectionManager.java:424) at org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.stop(ReplicationLogCleaner.java:141) at org.apache.hadoop.hbase.master.cleaner.CleanerChore.cleanup(CleanerChore.java:276) at org.apache.hadoop.hbase.Chore.run(Chore.java:94) at java.lang.Thread.run(Thread.java:745) 2014-08-24 06:15:47,410 INFO [M:1;ip-10-229-0-82:57826] master.HMaster(641): HMaster main thread exiting 2014-08-24 06:15:47,411 INFO [main] zookeeper.MiniZooKeeperCluster(246): Shutdown MiniZK cluster with all ZK servers 2014-08-24 06:15:47,411 WARN [main] datanode.DirectoryScanner(371): DirectoryScanner: shutdown has been called 2014-08-24 06:15:47,413 INFO [main] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0 2014-08-24 06:15:47,511 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(312): hconnection-0x47b642c5-0x14806a73b1d0014, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2014-08-24 06:15:47,511 DEBUG [RS:0;ip-10-229-0-82:57246-EventThread] zookeeper.ZooKeeperWatcher(312): hconnection-0x28f2994-0x14806a73b1d000d, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2014-08-24 06:15:47,511 DEBUG [RS:0;ip-10-229-0-82:57246-EventThread] zookeeper.ZooKeeperWatcher(394): hconnection-0x28f2994-0x14806a73b1d000d, quorum=localhost:56428, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2014-08-24 06:15:47,511 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(394): hconnection-0x47b642c5-0x14806a73b1d0014, quorum=localhost:56428, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2014-08-24 06:15:47,511 DEBUG [M:1;ip-10-229-0-82:57826-EventThread] zookeeper.ZooKeeperWatcher(312): hconnection-0x73c6f5-0x14806a73b1d001c, quorum=localhost:56428, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2014-08-24 06:15:47,511 DEBUG [M:1;ip-10-229-0-82:57826-EventThread] zookeeper.ZooKeeperWatcher(394): hconnection-0x73c6f5-0x14806a73b1d001c, quorum=localhost:56428, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2014-08-24 06:15:47,516 WARN [DataNode: [file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data7,file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data8] heartbeating to localhost/127.0.0.1:59365] datanode.BPServiceActor(575): BPOfferService for Block pool BP-799066276-10.229.0.82-1408860894595 (storage id DS-1725977906-10.229.0.82-45654-1408860895767) service to localhost/127.0.0.1:59365 interrupted 2014-08-24 06:15:47,516 WARN [DataNode: [file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data7,file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data8] heartbeating to localhost/127.0.0.1:59365] datanode.BPServiceActor(685): Ending block pool service for: Block pool BP-799066276-10.229.0.82-1408860894595 (storage id DS-1725977906-10.229.0.82-45654-1408860895767) service to localhost/127.0.0.1:59365 2014-08-24 06:15:47,516 WARN [main] datanode.DirectoryScanner(371): DirectoryScanner: shutdown has been called 2014-08-24 06:15:47,518 INFO [main] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0 2014-08-24 06:15:47,620 WARN [DataNode: [file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data5,file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data6] heartbeating to localhost/127.0.0.1:59365] datanode.BPServiceActor(575): BPOfferService for Block pool BP-799066276-10.229.0.82-1408860894595 (storage id DS-1729049550-10.229.0.82-42916-1408860895670) service to localhost/127.0.0.1:59365 interrupted 2014-08-24 06:15:47,620 WARN [DataNode: [file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data5,file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data6] heartbeating to localhost/127.0.0.1:59365] datanode.BPServiceActor(685): Ending block pool service for: Block pool BP-799066276-10.229.0.82-1408860894595 (storage id DS-1729049550-10.229.0.82-42916-1408860895670) service to localhost/127.0.0.1:59365 2014-08-24 06:15:47,621 WARN [main] datanode.DirectoryScanner(371): DirectoryScanner: shutdown has been called 2014-08-24 06:15:47,622 INFO [main] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0 2014-08-24 06:15:47,737 WARN [DataNode: [file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data3,file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data4] heartbeating to localhost/127.0.0.1:59365] datanode.BPServiceActor(575): BPOfferService for Block pool BP-799066276-10.229.0.82-1408860894595 (storage id DS-1150413765-10.229.0.82-60268-1408860895663) service to localhost/127.0.0.1:59365 interrupted 2014-08-24 06:15:47,737 WARN [DataNode: [file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data3,file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data4] heartbeating to localhost/127.0.0.1:59365] datanode.BPServiceActor(685): Ending block pool service for: Block pool BP-799066276-10.229.0.82-1408860894595 (storage id DS-1150413765-10.229.0.82-60268-1408860895663) service to localhost/127.0.0.1:59365 2014-08-24 06:15:47,738 WARN [main] datanode.DirectoryScanner(371): DirectoryScanner: shutdown has been called 2014-08-24 06:15:47,739 INFO [main] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0 2014-08-24 06:15:47,842 WARN [DataNode: [file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data1,file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data2] heartbeating to localhost/127.0.0.1:59365] datanode.BPServiceActor(575): BPOfferService for Block pool BP-799066276-10.229.0.82-1408860894595 (storage id DS-1350430660-10.229.0.82-39258-1408860895663) service to localhost/127.0.0.1:59365 interrupted 2014-08-24 06:15:47,842 WARN [DataNode: [file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data1,file:/usr/src/hbase/hbase-server/target/test-data/325b1b9c-4743-481e-a79b-9ff8b32084a3/dfscluster_d687dc39-502c-49d1-b9a5-b9ecf0229276/dfs/data/data2] heartbeating to localhost/127.0.0.1:59365] datanode.BPServiceActor(685): Ending block pool service for: Block pool BP-799066276-10.229.0.82-1408860894595 (storage id DS-1350430660-10.229.0.82-39258-1408860895663) service to localhost/127.0.0.1:59365 2014-08-24 06:15:47,845 WARN [org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor@1a738e7a] blockmanagement.DecommissionManager$Monitor(78): Monitor interrupted: java.lang.InterruptedException: sleep interrupted 2014-08-24 06:15:47,851 INFO [main] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0 2014-08-24 06:15:47,969 INFO [main] hbase.HBaseTestingUtility(961): Minicluster is down Tests run: 15, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 54.861 sec <<< FAILURE! 2014-08-24 06:15:47,971 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@35fc8e8a 2014-08-24 06:15:47,971 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2014-08-24 06:15:47,971 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@35fc8e8a 2014-08-24 06:15:47,971 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2014-08-24 06:15:47,971 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@35fc8e8a 2014-08-24 06:15:47,971 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2014-08-24 06:15:47,971 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@35fc8e8a 2014-08-24 06:15:47,971 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(120): Starting fs shutdown hook thread. 2014-08-24 06:15:47,973 ERROR [Thread-981] hdfs.DFSClient(776): Failed to close file /user/apurtell/hbase/data/default/testOpenFailedUnrecoverable/7c9369c4e1f8dc2024aa230a8f5fdbfa java.io.IOException: Failed on local exception: java.io.EOFException; Host Details : local host is: "ip-10-229-0-82.us-west-2.compute.internal/10.229.0.82"; destination host is: "localhost":59365; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) at org.apache.hadoop.ipc.Client.call(Client.java:1351) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy15.complete(Unknown Source) at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy15.complete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:371) at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:294) at com.sun.proxy.$Proxy20.complete(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:1894) at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1881) at org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:773) at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:790) at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:847) at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2524) at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2541) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891) 2014-08-24 06:15:47,974 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. Results : Failed tests: testCloseHang(org.apache.hadoop.hbase.master.TestZKLessAMOnCluster): expected: but was: Tests run: 15, Failures: 1, Errors: 0, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] HBase ............................................. SUCCESS [ 1.373 s] [INFO] HBase - Common .................................... SUCCESS [ 3.205 s] [INFO] HBase - Protocol .................................. SUCCESS [ 0.081 s] [INFO] HBase - Client .................................... SUCCESS [ 0.679 s] [INFO] HBase - Hadoop Compatibility ...................... SUCCESS [ 0.073 s] [INFO] HBase - Hadoop Two Compatibility .................. SUCCESS [ 0.563 s] [INFO] HBase - Prefix Tree ............................... SUCCESS [ 0.356 s] [INFO] HBase - Server .................................... FAILURE [ 56.967 s] [INFO] HBase - Testing Util .............................. SKIPPED [INFO] HBase - Thrift .................................... SKIPPED [INFO] HBase - Shell ..................................... SKIPPED [INFO] HBase - Integration Tests ......................... SKIPPED [INFO] HBase - Examples .................................. SKIPPED [INFO] HBase - Assembly .................................. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:03 min [INFO] Finished at: 2014-08-24T06:15:48+00:00 [INFO] Final Memory: 53M/812M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12-TRUNK-HBASE-2:test (default-test) on project hbase-server: There are test failures. [ERROR] [ERROR] Please refer to /usr/src/hbase/hbase-server/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hbase-server