2015-04-04 10:55:35,375 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testLegacyRecovery Thread=368, OpenFileDescriptor=424, MaxFileDescriptor=10240, SystemLoadAverage=285, ProcessCount=236, AvailableMemoryMB=17, ConnectionCount=1 2015-04-04 10:55:35,376 INFO [Thread-346] coprocessor.TestRegionObserverInterface(648): org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testLegacyRecovery 2015-04-04 10:55:35,378 INFO [B.defaultRpcServer.handler=3,queue=0,port=52158] master.HMaster(1318): Client=apurtell/null create 'TestTable.testLegacyRecovery', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:55:35,382 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testLegacyRecovery/write-master:521580000000000 2015-04-04 10:55:35,390 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(188): Create table TestTable.testLegacyRecovery 2015-04-04 10:55:35,399 INFO [IPC Server handler 0 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741851_1027{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 792 2015-04-04 10:55:35,804 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testLegacyRecovery/.tabledesc/.tableinfo.0000000001 2015-04-04 10:55:35,807 INFO [RegionOpenAndInitThread-TestTable.testLegacyRecovery-1] regionserver.HRegion(5885): creating HRegion TestTable.testLegacyRecovery HTD == 'TestTable.testLegacyRecovery', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp Table name == TestTable.testLegacyRecovery 2015-04-04 10:55:35,816 INFO [IPC Server handler 2 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741852_1028{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 63 2015-04-04 10:55:36,222 DEBUG [RegionOpenAndInitThread-TestTable.testLegacyRecovery-1] regionserver.HRegion(721): Instantiated TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,223 DEBUG [RegionOpenAndInitThread-TestTable.testLegacyRecovery-1] regionserver.HRegion(1350): Closing TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.: disabling compactions & flushes 2015-04-04 10:55:36,223 DEBUG [RegionOpenAndInitThread-TestTable.testLegacyRecovery-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,223 INFO [RegionOpenAndInitThread-TestTable.testLegacyRecovery-1] regionserver.HRegion(1471): Closed TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,228 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:55:36,229 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,52161,1428170104473 2015-04-04 10:55:36,229 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKAssign(174): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Async create of unassigned node 5bcaf4310f0272d29522a5c3ece41cf2 with OFFLINE state 2015-04-04 10:55:36,230 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:36,230 DEBUG [main-EventThread] master.OfflineCallback(69): rs={5bcaf4310f0272d29522a5c3ece41cf2 state=OFFLINE, ts=1428170136229, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:36,231 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={5bcaf4310f0272d29522a5c3ece41cf2 state=OFFLINE, ts=1428170136229, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:36,236 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1658): localhost,52161,1428170104473 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:55:36,236 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.RegionStates(1112): Transition {5bcaf4310f0272d29522a5c3ece41cf2 state=OFFLINE, ts=1428170136229, server=null} to {5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_OPEN, ts=1428170136236, server=localhost,52161,1428170104473} 2015-04-04 10:55:36,237 INFO [PriorityRpcServer.handler=4,queue=0,port=52161] regionserver.RSRpcServices(1390): Open TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,245 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 5bcaf4310f0272d29522a5c3ece41cf2 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:36,245 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1788): Bulk assigning done for localhost,52161,1428170104473 2015-04-04 10:55:36,245 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testLegacyRecovery state from ENABLING to ENABLED 2015-04-04 10:55:36,247 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,247 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 5bcaf4310f0272d29522a5c3ece41cf2 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:36,247 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(6078): Opening region: {ENCODED => 5bcaf4310f0272d29522a5c3ece41cf2, NAME => 'TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:36,248 DEBUG [AM.ZK.Worker-pool3-t11] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,52161,1428170104473, region=5bcaf4310f0272d29522a5c3ece41cf2, current_state={5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_OPEN, ts=1428170136236, server=localhost,52161,1428170104473} 2015-04-04 10:55:36,248 INFO [AM.ZK.Worker-pool3-t11] master.RegionStates(1112): Transition {5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_OPEN, ts=1428170136236, server=localhost,52161,1428170104473} to {5bcaf4310f0272d29522a5c3ece41cf2 state=OPENING, ts=1428170136248, server=localhost,52161,1428170104473} 2015-04-04 10:55:36,248 DEBUG [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:36,248 DEBUG [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:36,248 INFO [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:55:36,248 INFO [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:55:36,249 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testLegacyRecovery 5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,249 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(721): Instantiated TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,252 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testLegacyRecovery/write-master:521580000000000 2015-04-04 10:55:36,253 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(218): Table, TestTable.testLegacyRecovery, creation successful 2015-04-04 10:55:36,254 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:36,254 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:36,256 DEBUG [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/a 2015-04-04 10:55:36,258 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:36,258 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:36,259 DEBUG [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/b 2015-04-04 10:55:36,261 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:36,262 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:36,263 DEBUG [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/c 2015-04-04 10:55:36,264 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,267 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:55:36,271 DEBUG [RS_OPEN_REGION-localhost:52161-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:55:36,271 INFO [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(847): Onlined 5bcaf4310f0272d29522a5c3ece41cf2; next sequenceid=2 2015-04-04 10:55:36,272 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(643): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Attempting to retransition opening state of node 5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,273 INFO [PostOpenDeployTasks:5bcaf4310f0272d29522a5c3ece41cf2] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,277 INFO [PostOpenDeployTasks:5bcaf4310f0272d29522a5c3ece41cf2] hbase.MetaTableAccessor(1350): Updated row TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. with server=localhost,52161,1428170104473 2015-04-04 10:55:36,277 DEBUG [PostOpenDeployTasks:5bcaf4310f0272d29522a5c3ece41cf2] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,279 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 5bcaf4310f0272d29522a5c3ece41cf2 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:36,281 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,281 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 5bcaf4310f0272d29522a5c3ece41cf2 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:36,282 DEBUG [RS_OPEN_REGION-localhost:52161-0] coordination.ZkOpenRegionCoordination(91): Transitioned 5bcaf4310f0272d29522a5c3ece41cf2 to OPENED in zk on localhost,52161,1428170104473 2015-04-04 10:55:36,282 DEBUG [RS_OPEN_REGION-localhost:52161-0] handler.OpenRegionHandler(167): Opened TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. on localhost,52161,1428170104473 2015-04-04 10:55:36,282 DEBUG [AM.ZK.Worker-pool3-t10] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,52161,1428170104473, region=5bcaf4310f0272d29522a5c3ece41cf2, current_state={5bcaf4310f0272d29522a5c3ece41cf2 state=OPENING, ts=1428170136248, server=localhost,52161,1428170104473} 2015-04-04 10:55:36,282 INFO [AM.ZK.Worker-pool3-t10] master.RegionStates(1112): Transition {5bcaf4310f0272d29522a5c3ece41cf2 state=OPENING, ts=1428170136248, server=localhost,52161,1428170104473} to {5bcaf4310f0272d29522a5c3ece41cf2 state=OPEN, ts=1428170136282, server=localhost,52161,1428170104473} 2015-04-04 10:55:36,282 DEBUG [AM.ZK.Worker-pool3-t10] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 5bcaf4310f0272d29522a5c3ece41cf2 from localhost,52158,1428170104089; deleting unassigned node 2015-04-04 10:55:36,284 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,284 DEBUG [AM.ZK.Worker-pool3-t10] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 5bcaf4310f0272d29522a5c3ece41cf2 in expected state RS_ZK_REGION_OPENED 2015-04-04 10:55:36,284 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:36,284 DEBUG [AM.ZK.Worker-pool3-t10] master.AssignmentManager$4(1337): Znode TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. deleted, state: {5bcaf4310f0272d29522a5c3ece41cf2 state=OPEN, ts=1428170136282, server=localhost,52161,1428170104473} 2015-04-04 10:55:36,284 DEBUG [AM.ZK.Worker-pool3-t10] master.RegionStates(460): Onlined 5bcaf4310f0272d29522a5c3ece41cf2 on localhost,52161,1428170104473 {ENCODED => 5bcaf4310f0272d29522a5c3ece41cf2, NAME => 'TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:36,532 INFO [Thread-346] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:36,539 INFO [Thread-346] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:36,566 INFO [Thread-346] client.ConnectionUtils(103): regionserver/localhost/127.0.0.1:0 server-side HConnection retries=350 2015-04-04 10:55:36,566 INFO [Thread-346] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-04-04 10:55:36,569 INFO [Thread-346] ipc.RpcServer$Listener(587): regionserver/localhost/127.0.0.1:0: started 10 reader(s). 2015-04-04 10:55:36,571 INFO [Thread-346] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-04-04 10:55:36,574 INFO [Thread-346] zookeeper.RecoverableZooKeeper(121): Process identifier=regionserver:52251 connecting to ZooKeeper ensemble=localhost:49474 2015-04-04 10:55:36,578 DEBUG [Thread-346-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:522510x0, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-04-04 10:55:36,578 DEBUG [Thread-346] zookeeper.ZKUtil(490): regionserver:522510x0, quorum=localhost:49474, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-04-04 10:55:36,580 DEBUG [Thread-346-EventThread] zookeeper.ZooKeeperWatcher(485): regionserver:52251-0x14c859259d50016 connected 2015-04-04 10:55:36,580 DEBUG [Thread-346] zookeeper.ZKUtil(490): regionserver:52251-0x14c859259d50016, quorum=localhost:49474, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-04-04 10:55:36,581 INFO [RpcServer.responder] ipc.RpcServer$Responder(906): RpcServer.responder: starting 2015-04-04 10:55:36,581 DEBUG [Thread-346] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-04-04 10:55:36,581 INFO [RpcServer.listener,port=52251] ipc.RpcServer$Listener(718): RpcServer.listener,port=52251: starting 2015-04-04 10:55:36,581 DEBUG [Thread-346] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-04-04 10:55:36,581 DEBUG [Thread-346] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-04-04 10:55:36,582 DEBUG [Thread-346] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-04-04 10:55:36,582 DEBUG [Thread-346] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-04-04 10:55:36,582 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-04-04 10:55:36,582 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-04-04 10:55:36,583 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-04-04 10:55:36,583 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-04-04 10:55:36,583 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-04-04 10:55:36,583 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-04-04 10:55:36,584 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-04-04 10:55:36,584 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-04-04 10:55:36,584 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-04-04 10:55:36,585 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-04-04 10:55:36,585 DEBUG [Thread-346] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-04-04 10:55:36,585 DEBUG [Thread-346] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-04-04 10:55:36,585 DEBUG [Thread-346] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-04-04 10:55:36,586 INFO [Thread-346] http.HttpRequestLog(69): Http request log for http.requests.regionserver is not defined 2015-04-04 10:55:36,587 INFO [Thread-346] http.HttpServer(819): Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2015-04-04 10:55:36,587 INFO [Thread-346] http.HttpServer(797): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver 2015-04-04 10:55:36,588 INFO [Thread-346] http.HttpServer(804): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2015-04-04 10:55:36,588 INFO [Thread-346] http.HttpServer(804): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2015-04-04 10:55:36,590 INFO [Thread-346] http.HttpServer(1011): Jetty bound to port 52253 2015-04-04 10:55:36,590 INFO [Thread-346] log.Slf4jLog(67): jetty-6.1.26 2015-04-04 10:55:36,610 INFO [Thread-346] log.Slf4jLog(67): Started SelectChannelConnector@0.0.0.0:52253 2015-04-04 10:55:36,615 INFO [RS:3;localhost:52251] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x21664e5f connecting to ZooKeeper ensemble=localhost:49474 2015-04-04 10:55:36,618 DEBUG [RS:3;localhost:52251-EventThread] zookeeper.ZooKeeperWatcher(401): hconnection-0x21664e5f0x0, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-04-04 10:55:36,619 DEBUG [RS:3;localhost:52251] ipc.AbstractRpcClient(105): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2212f7a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-04-04 10:55:36,620 DEBUG [RS:3;localhost:52251-EventThread] zookeeper.ZooKeeperWatcher(485): hconnection-0x21664e5f-0x14c859259d50017 connected 2015-04-04 10:55:36,621 INFO [RS:3;localhost:52251] regionserver.HRegionServer(760): ClusterId : 81c27d3e-db00-4d35-891c-9756f3546cb2 2015-04-04 10:55:36,621 DEBUG [RS:3;localhost:52251] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-04-04 10:55:36,622 DEBUG [RS:3;localhost:52251] zookeeper.RecoverableZooKeeper(531): Node /hbase/online-snapshot/acquired already exists 2015-04-04 10:55:36,623 DEBUG [RS:3;localhost:52251] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-04-04 10:55:36,623 DEBUG [RS:3;localhost:52251] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-04-04 10:55:36,624 DEBUG [RS:3;localhost:52251] zookeeper.RecoverableZooKeeper(531): Node /hbase/flush-table-proc/acquired already exists 2015-04-04 10:55:36,625 DEBUG [RS:3;localhost:52251] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-04-04 10:55:36,625 INFO [RS:3;localhost:52251] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-04-04 10:55:36,625 INFO [RS:3;localhost:52251] regionserver.HRegionServer$CompactionChecker(1477): CompactionChecker runs every 1sec 2015-04-04 10:55:36,625 DEBUG [RS:3;localhost:52251] ipc.AbstractRpcClient(105): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6efa3ea4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=localhost/127.0.0.1:0 2015-04-04 10:55:36,626 DEBUG [RS:3;localhost:52251] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:RS:3;localhost:52251 2015-04-04 10:55:36,627 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-04-04 10:55:36,627 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-04-04 10:55:36,627 DEBUG [RS:3;localhost:52251] zookeeper.ZKUtil(490): regionserver:52251-0x14c859259d50016, quorum=localhost:49474, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,52251,1428170136570 2015-04-04 10:55:36,628 INFO [RS:3;localhost:52251] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-04-04 10:55:36,628 INFO [RS:3;localhost:52251] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-04-04 10:55:36,628 DEBUG [main-EventThread] zookeeper.ZKUtil(490): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,52251,1428170136570 2015-04-04 10:55:36,628 DEBUG [main-EventThread] zookeeper.ZKUtil(490): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,52251,1428170136570 2015-04-04 10:55:36,628 INFO [RS:3;localhost:52251] regionserver.HRegionServer(2196): reportForDuty to master=localhost,52158,1428170104089 with port=52251, startcode=1428170136570 2015-04-04 10:55:36,628 DEBUG [RS:3;localhost:52251] ipc.RpcClientImpl$Connection(343): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-04-04 10:55:36,629 DEBUG [RS:3;localhost:52251] ipc.RpcClientImpl$Connection(708): Connecting to localhost/127.0.0.1:52158 2015-04-04 10:55:36,629 DEBUG [main-EventThread] zookeeper.ZKUtil(490): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,52161,1428170104473 2015-04-04 10:55:36,629 DEBUG [main-EventThread] zookeeper.ZKUtil(490): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,52161,1428170104473 2015-04-04 10:55:36,630 DEBUG [RpcServer.listener,port=52158] ipc.RpcServer$Listener(829): RpcServer.listener,port=52158: connection from 127.0.0.1:52255; # active connections: 3 2015-04-04 10:55:36,631 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/localhost,52251,1428170136570 2015-04-04 10:55:36,631 INFO [RpcServer.reader=5,bindAddress=localhost,port=52158] ipc.RpcServer$Connection(1661): Connection from 127.0.0.1 port: 52255 with version info: version: "1.1.0-SNAPSHOT" url: "git://apurtell-ltm2.internal.salesforce.com/Users/apurtell/src/hbase" revision: "5c19f9eac7d1f22147345f09aad2f04fec1ad67d" user: "apurtell" date: "Sat Apr 4 10:54:41 PDT 2015" src_checksum: "2a921de567e8e6d9e1c358f3276c1ec9" 2015-04-04 10:55:36,631 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/localhost,52161,1428170104473 2015-04-04 10:55:36,632 INFO [B.defaultRpcServer.handler=0,queue=0,port=52158] master.ServerManager(448): Registering server=localhost,52251,1428170136570 2015-04-04 10:55:36,632 INFO [RS:3;localhost:52251] regionserver.HRegionServer(1315): Config from master: hbase.rootdir=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333 2015-04-04 10:55:36,633 INFO [RS:3;localhost:52251] regionserver.HRegionServer(1315): Config from master: fs.defaultFS=hdfs://localhost:52150 2015-04-04 10:55:36,633 INFO [RS:3;localhost:52251] regionserver.HRegionServer(1315): Config from master: hbase.master.info.port=52160 2015-04-04 10:55:36,633 WARN [RS:3;localhost:52251] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-04-04 10:55:36,633 INFO [RS:3;localhost:52251] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:36,633 DEBUG [RS:3;localhost:52251] regionserver.HRegionServer(1574): logdir=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/WALs/localhost,52251,1428170136570 2015-04-04 10:55:36,640 DEBUG [RS:3;localhost:52251] regionserver.Replication(143): ReplicationStatisticsThread 300 2015-04-04 10:55:36,640 INFO [RS:3;localhost:52251] wal.WALFactory(144): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-04-04 10:55:36,645 INFO [RS:3;localhost:52251] wal.FSHLog(550): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=localhost%2C52251%2C1428170136570.default, suffix=, logDir=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/WALs/localhost,52251,1428170136570, archiveDir=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/oldWALs 2015-04-04 10:55:36,655 INFO [RS:3;localhost:52251] wal.FSHLog(1495): Slow sync cost: 7 ms, current pipeline: [] 2015-04-04 10:55:36,655 INFO [RS:3;localhost:52251] wal.FSHLog(972): New WAL /user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/WALs/localhost,52251,1428170136570/localhost%2C52251%2C1428170136570.default.1428170136645 2015-04-04 10:55:36,656 INFO [RS:3;localhost:52251] regionserver.MetricsRegionServerWrapperImpl(100): Computing regionserver metrics every 5000 milliseconds 2015-04-04 10:55:36,658 DEBUG [RS:3;localhost:52251] executor.ExecutorService(99): Starting executor service name=RS_OPEN_REGION-localhost:52251, corePoolSize=3, maxPoolSize=3 2015-04-04 10:55:36,658 DEBUG [RS:3;localhost:52251] executor.ExecutorService(99): Starting executor service name=RS_OPEN_META-localhost:52251, corePoolSize=1, maxPoolSize=1 2015-04-04 10:55:36,658 DEBUG [RS:3;localhost:52251] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_REGION-localhost:52251, corePoolSize=3, maxPoolSize=3 2015-04-04 10:55:36,658 DEBUG [RS:3;localhost:52251] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_META-localhost:52251, corePoolSize=1, maxPoolSize=1 2015-04-04 10:55:36,658 DEBUG [RS:3;localhost:52251] executor.ExecutorService(99): Starting executor service name=RS_LOG_REPLAY_OPS-localhost:52251, corePoolSize=2, maxPoolSize=2 2015-04-04 10:55:36,658 DEBUG [RS:3;localhost:52251] executor.ExecutorService(99): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-localhost:52251, corePoolSize=3, maxPoolSize=3 2015-04-04 10:55:36,663 DEBUG [RS:3;localhost:52251] zookeeper.ZKUtil(490): regionserver:52251-0x14c859259d50016, quorum=localhost:49474, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,52251,1428170136570 2015-04-04 10:55:36,663 DEBUG [RS:3;localhost:52251] zookeeper.ZKUtil(490): regionserver:52251-0x14c859259d50016, quorum=localhost:49474, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,52161,1428170104473 2015-04-04 10:55:36,664 INFO [RS:3;localhost:52251] regionserver.ReplicationSourceManager(227): Current list of replicators: [localhost,52202,1428170110438, localhost,52251,1428170136570, localhost,52161,1428170104473, localhost,52230,1428170132721] other RSs: [localhost,52251,1428170136570, localhost,52161,1428170104473] 2015-04-04 10:55:36,686 INFO [RS:3;localhost:52251] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x21058c14 connecting to ZooKeeper ensemble=localhost:49474 2015-04-04 10:55:36,689 DEBUG [RS:3;localhost:52251-EventThread] zookeeper.ZooKeeperWatcher(401): hconnection-0x21058c140x0, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-04-04 10:55:36,690 DEBUG [RS:3;localhost:52251] ipc.AbstractRpcClient(105): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@50d8ffd1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-04-04 10:55:36,691 DEBUG [RS:3;localhost:52251-EventThread] zookeeper.ZooKeeperWatcher(485): hconnection-0x21058c14-0x14c859259d50018 connected 2015-04-04 10:55:36,710 INFO [RS:3;localhost:52251] regionserver.HeapMemoryManager(187): Starting HeapMemoryTuner chore. 2015-04-04 10:55:36,710 INFO [SplitLogWorker-localhost:52251] regionserver.SplitLogWorker(134): SplitLogWorker localhost,52251,1428170136570 starting 2015-04-04 10:55:36,710 INFO [RS:3;localhost:52251] regionserver.HRegionServer(1337): Serving as localhost,52251,1428170136570, RpcServer on localhost/127.0.0.1:52251, sessionid=0x14c859259d50016 2015-04-04 10:55:36,710 DEBUG [RS:3;localhost:52251] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-04-04 10:55:36,710 DEBUG [RS:3;localhost:52251] snapshot.RegionServerSnapshotManager(122): Start Snapshot Manager localhost,52251,1428170136570 2015-04-04 10:55:36,710 DEBUG [RS:3;localhost:52251] procedure.ZKProcedureMemberRpcs(349): Starting procedure member 'localhost,52251,1428170136570' 2015-04-04 10:55:36,710 DEBUG [RS:3;localhost:52251] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-04-04 10:55:36,711 DEBUG [RS:3;localhost:52251] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-04-04 10:55:36,712 DEBUG [RS:3;localhost:52251] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-04-04 10:55:36,712 DEBUG [RS:3;localhost:52251] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-04-04 10:55:36,712 DEBUG [RS:3;localhost:52251] flush.RegionServerFlushTableProcedureManager(101): Start region server flush procedure manager localhost,52251,1428170136570 2015-04-04 10:55:36,712 DEBUG [RS:3;localhost:52251] procedure.ZKProcedureMemberRpcs(349): Starting procedure member 'localhost,52251,1428170136570' 2015-04-04 10:55:36,712 DEBUG [RS:3;localhost:52251] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-04-04 10:55:36,712 DEBUG [RS:3;localhost:52251] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-04-04 10:55:36,713 DEBUG [RS:3;localhost:52251] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-04-04 10:55:36,716 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=52158] master.ServerManager(951): New admin connection to localhost,52251,1428170136570 2015-04-04 10:55:36,716 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=52158] ipc.RpcClientImpl$Connection(343): Use SIMPLE authentication for service AdminService, sasl=false 2015-04-04 10:55:36,717 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=52158] ipc.RpcClientImpl$Connection(708): Connecting to localhost/127.0.0.1:52251 2015-04-04 10:55:36,717 DEBUG [RpcServer.listener,port=52251] ipc.RpcServer$Listener(829): RpcServer.listener,port=52251: connection from 127.0.0.1:52259; # active connections: 1 2015-04-04 10:55:36,719 INFO [RpcServer.reader=1,bindAddress=localhost,port=52251] ipc.RpcServer$Connection(1661): Connection from 127.0.0.1 port: 52259 with version info: version: "1.1.0-SNAPSHOT" url: "git://apurtell-ltm2.internal.salesforce.com/Users/apurtell/src/hbase" revision: "5c19f9eac7d1f22147345f09aad2f04fec1ad67d" user: "apurtell" date: "Sat Apr 4 10:54:41 PDT 2015" src_checksum: "2a921de567e8e6d9e1c358f3276c1ec9" 2015-04-04 10:55:36,720 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52251] regionserver.RSRpcServices(1498): Warming up Region TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,728 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52251] regionserver.HRegion(6141): HRegion.Warming up region: {ENCODED => 5bcaf4310f0272d29522a5c3ece41cf2, NAME => 'TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:36,728 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52251] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:36,728 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52251] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:36,729 INFO [B.defaultRpcServer.handler=0,queue=0,port=52251] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:55:36,729 INFO [B.defaultRpcServer.handler=0,queue=0,port=52251] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:55:36,729 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52251] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testLegacyRecovery 5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,729 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52251] regionserver.HRegion(721): Instantiated TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,732 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:36,732 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:36,733 DEBUG [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/a 2015-04-04 10:55:36,735 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:36,735 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:36,737 DEBUG [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/b 2015-04-04 10:55:36,739 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:36,739 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:36,741 DEBUG [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/c 2015-04-04 10:55:36,741 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52251] regionserver.HRegion(1350): Closing TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.: disabling compactions & flushes 2015-04-04 10:55:36,741 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52251] regionserver.HRegion(1377): Updates disabled for region TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,742 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.-1] regionserver.HStore(879): Closed a 2015-04-04 10:55:36,742 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.-1] regionserver.HStore(879): Closed b 2015-04-04 10:55:36,742 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.-1] regionserver.HStore(879): Closed c 2015-04-04 10:55:36,747 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52251] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:55:36,747 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52251] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:55:36,747 INFO [B.defaultRpcServer.handler=0,queue=0,port=52251] regionserver.HRegion(1471): Closed TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,747 INFO [B.defaultRpcServer.handler=3,queue=0,port=52158] master.HMaster(1289): Client=apurtell/null move hri=TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2., src=localhost,52161,1428170104473, dest=localhost,52251,1428170136570, running balancer 2015-04-04 10:55:36,748 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=52158] master.AssignmentManager(2452): Starting unassign of TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. (offlining), current state: {5bcaf4310f0272d29522a5c3ece41cf2 state=OPEN, ts=1428170136284, server=localhost,52161,1428170104473} 2015-04-04 10:55:36,748 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=52158] zookeeper.ZKAssign(527): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Creating unassigned node 5bcaf4310f0272d29522a5c3ece41cf2 in a CLOSING state 2015-04-04 10:55:36,749 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:36,749 INFO [B.defaultRpcServer.handler=3,queue=0,port=52158] master.RegionStates(1112): Transition {5bcaf4310f0272d29522a5c3ece41cf2 state=OPEN, ts=1428170136284, server=localhost,52161,1428170104473} to {5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_CLOSE, ts=1428170136749, server=localhost,52161,1428170104473} 2015-04-04 10:55:36,750 INFO [PriorityRpcServer.handler=5,queue=1,port=52161] regionserver.RSRpcServices(1018): Close 5bcaf4310f0272d29522a5c3ece41cf2, moving to localhost,52251,1428170136570 2015-04-04 10:55:36,750 DEBUG [RS_CLOSE_REGION-localhost:52161-0] handler.CloseRegionHandler(119): Processing close of TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,750 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=52158] master.AssignmentManager(1836): Sent CLOSE to localhost,52161,1428170104473 for region TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,751 DEBUG [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(1350): Closing TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.: disabling compactions & flushes 2015-04-04 10:55:36,751 DEBUG [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(1377): Updates disabled for region TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,751 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.-1] regionserver.HStore(879): Closed a 2015-04-04 10:55:36,752 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.-1] regionserver.HStore(879): Closed b 2015-04-04 10:55:36,752 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.-1] regionserver.HStore(879): Closed c 2015-04-04 10:55:36,758 DEBUG [RS_CLOSE_REGION-localhost:52161-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits/4.seqid to file, newSeqId=4, maxSeqId=2 2015-04-04 10:55:36,759 DEBUG [RS_CLOSE_REGION-localhost:52161-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:55:36,759 DEBUG [RS_CLOSE_REGION-localhost:52161-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:55:36,759 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(1471): Closed TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,760 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegionServer(2951): Adding moved region record: 5bcaf4310f0272d29522a5c3ece41cf2 to localhost,52251,1428170136570 as of 2 2015-04-04 10:55:36,760 DEBUG [RS_CLOSE_REGION-localhost:52161-0] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 5bcaf4310f0272d29522a5c3ece41cf2 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:36,761 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,761 DEBUG [RS_CLOSE_REGION-localhost:52161-0] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 5bcaf4310f0272d29522a5c3ece41cf2 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:36,762 DEBUG [RS_CLOSE_REGION-localhost:52161-0] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. on localhost,52161,1428170104473 2015-04-04 10:55:36,762 DEBUG [RS_CLOSE_REGION-localhost:52161-0] handler.CloseRegionHandler(164): Closed TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,762 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,52161,1428170104473, region=5bcaf4310f0272d29522a5c3ece41cf2, current_state={5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_CLOSE, ts=1428170136749, server=localhost,52161,1428170104473} 2015-04-04 10:55:36,762 DEBUG [AM.ZK.Worker-pool3-t14] handler.ClosedRegionHandler(93): Handling CLOSED event for 5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,762 INFO [AM.ZK.Worker-pool3-t14] master.RegionStates(1112): Transition {5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_CLOSE, ts=1428170136749, server=localhost,52161,1428170104473} to {5bcaf4310f0272d29522a5c3ece41cf2 state=CLOSED, ts=1428170136762, server=localhost,52161,1428170104473} 2015-04-04 10:55:36,762 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(2363): Found an existing plan for TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. destination server is localhost,52251,1428170136570 accepted as a dest server = true 2015-04-04 10:55:36,762 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(2401): Using pre-existing plan for TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.; plan=hri=TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2., src=localhost,52161,1428170104473, dest=localhost,52251,1428170136570 2015-04-04 10:55:36,762 INFO [AM.ZK.Worker-pool3-t14] master.RegionStates(1112): Transition {5bcaf4310f0272d29522a5c3ece41cf2 state=CLOSED, ts=1428170136762, server=localhost,52161,1428170104473} to {5bcaf4310f0272d29522a5c3ece41cf2 state=OFFLINE, ts=1428170136762, server=localhost,52161,1428170104473} 2015-04-04 10:55:36,763 DEBUG [AM.ZK.Worker-pool3-t14] zookeeper.ZKAssign(205): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Creating (or updating) unassigned node 5bcaf4310f0272d29522a5c3ece41cf2 with OFFLINE state 2015-04-04 10:55:36,764 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,764 INFO [AM.ZK.Worker-pool3-t14] master.AssignmentManager(2104): Assigning TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. to localhost,52251,1428170136570 2015-04-04 10:55:36,764 INFO [AM.ZK.Worker-pool3-t14] master.RegionStates(1112): Transition {5bcaf4310f0272d29522a5c3ece41cf2 state=OFFLINE, ts=1428170136762, server=localhost,52161,1428170104473} to {5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_OPEN, ts=1428170136764, server=localhost,52251,1428170136570} 2015-04-04 10:55:36,765 INFO [PriorityRpcServer.handler=0,queue=0,port=52251] regionserver.RSRpcServices(1390): Open TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,771 DEBUG [RS_OPEN_REGION-localhost:52251-0] zookeeper.ZKAssign(805): regionserver:52251-0x14c859259d50016, quorum=localhost:49474, baseZNode=/hbase Transitioning 5bcaf4310f0272d29522a5c3ece41cf2 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:36,772 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(948): Handling M_ZK_REGION_OFFLINE, server=localhost,52251,1428170136570, region=5bcaf4310f0272d29522a5c3ece41cf2, current_state={5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_OPEN, ts=1428170136764, server=localhost,52251,1428170136570} 2015-04-04 10:55:36,773 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,773 DEBUG [RS_OPEN_REGION-localhost:52251-0] zookeeper.ZKAssign(880): regionserver:52251-0x14c859259d50016, quorum=localhost:49474, baseZNode=/hbase Transitioned node 5bcaf4310f0272d29522a5c3ece41cf2 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:36,773 DEBUG [RS_OPEN_REGION-localhost:52251-0] regionserver.HRegion(6078): Opening region: {ENCODED => 5bcaf4310f0272d29522a5c3ece41cf2, NAME => 'TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:36,774 DEBUG [AM.ZK.Worker-pool3-t19] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,52251,1428170136570, region=5bcaf4310f0272d29522a5c3ece41cf2, current_state={5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_OPEN, ts=1428170136764, server=localhost,52251,1428170136570} 2015-04-04 10:55:36,774 INFO [AM.ZK.Worker-pool3-t19] master.RegionStates(1112): Transition {5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_OPEN, ts=1428170136764, server=localhost,52251,1428170136570} to {5bcaf4310f0272d29522a5c3ece41cf2 state=OPENING, ts=1428170136774, server=localhost,52251,1428170136570} 2015-04-04 10:55:36,774 DEBUG [RS_OPEN_REGION-localhost:52251-0] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:36,774 DEBUG [RS_OPEN_REGION-localhost:52251-0] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:36,774 INFO [RS_OPEN_REGION-localhost:52251-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:55:36,774 INFO [RS_OPEN_REGION-localhost:52251-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:55:36,774 DEBUG [RS_OPEN_REGION-localhost:52251-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testLegacyRecovery 5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,775 DEBUG [RS_OPEN_REGION-localhost:52251-0] regionserver.HRegion(721): Instantiated TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,778 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:36,778 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:36,780 DEBUG [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/a 2015-04-04 10:55:36,781 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:36,781 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:36,782 DEBUG [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/b 2015-04-04 10:55:36,783 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:36,784 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:36,785 DEBUG [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/c 2015-04-04 10:55:36,788 DEBUG [RS_OPEN_REGION-localhost:52251-0] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,790 DEBUG [RS_OPEN_REGION-localhost:52251-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:55:36,796 DEBUG [RS_OPEN_REGION-localhost:52251-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits/5.seqid to file, newSeqId=5, maxSeqId=4 2015-04-04 10:55:36,797 INFO [RS_OPEN_REGION-localhost:52251-0] regionserver.HRegion(847): Onlined 5bcaf4310f0272d29522a5c3ece41cf2; next sequenceid=5 2015-04-04 10:55:36,798 DEBUG [RS_OPEN_REGION-localhost:52251-0] zookeeper.ZKAssign(643): regionserver:52251-0x14c859259d50016, quorum=localhost:49474, baseZNode=/hbase Attempting to retransition opening state of node 5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,799 INFO [PostOpenDeployTasks:5bcaf4310f0272d29522a5c3ece41cf2] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,802 DEBUG [hconnection-0x21664e5f-shared--pool67-t1] ipc.RpcClientImpl$Connection(343): Use SIMPLE authentication for service ClientService, sasl=false 2015-04-04 10:55:36,802 DEBUG [hconnection-0x21664e5f-shared--pool67-t1] ipc.RpcClientImpl$Connection(708): Connecting to localhost/127.0.0.1:52161 2015-04-04 10:55:36,803 DEBUG [RpcServer.listener,port=52161] ipc.RpcServer$Listener(829): RpcServer.listener,port=52161: connection from 127.0.0.1:52260; # active connections: 6 2015-04-04 10:55:36,804 INFO [RpcServer.reader=7,bindAddress=localhost,port=52161] ipc.RpcServer$Connection(1661): Connection from 127.0.0.1 port: 52260 with version info: version: "1.1.0-SNAPSHOT" url: "git://apurtell-ltm2.internal.salesforce.com/Users/apurtell/src/hbase" revision: "5c19f9eac7d1f22147345f09aad2f04fec1ad67d" user: "apurtell" date: "Sat Apr 4 10:54:41 PDT 2015" src_checksum: "2a921de567e8e6d9e1c358f3276c1ec9" 2015-04-04 10:55:36,806 INFO [PostOpenDeployTasks:5bcaf4310f0272d29522a5c3ece41cf2] hbase.MetaTableAccessor(1350): Updated row TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. with server=localhost,52251,1428170136570 2015-04-04 10:55:36,806 DEBUG [PostOpenDeployTasks:5bcaf4310f0272d29522a5c3ece41cf2] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:36,807 DEBUG [RS_OPEN_REGION-localhost:52251-0] zookeeper.ZKAssign(805): regionserver:52251-0x14c859259d50016, quorum=localhost:49474, baseZNode=/hbase Transitioning 5bcaf4310f0272d29522a5c3ece41cf2 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:36,808 DEBUG [RS_OPEN_REGION-localhost:52251-0] zookeeper.ZKAssign(880): regionserver:52251-0x14c859259d50016, quorum=localhost:49474, baseZNode=/hbase Transitioned node 5bcaf4310f0272d29522a5c3ece41cf2 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:36,809 DEBUG [RS_OPEN_REGION-localhost:52251-0] coordination.ZkOpenRegionCoordination(91): Transitioned 5bcaf4310f0272d29522a5c3ece41cf2 to OPENED in zk on localhost,52251,1428170136570 2015-04-04 10:55:36,808 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,809 DEBUG [RS_OPEN_REGION-localhost:52251-0] handler.OpenRegionHandler(167): Opened TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. on localhost,52251,1428170136570 2015-04-04 10:55:36,810 DEBUG [AM.ZK.Worker-pool3-t15] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,52251,1428170136570, region=5bcaf4310f0272d29522a5c3ece41cf2, current_state={5bcaf4310f0272d29522a5c3ece41cf2 state=OPENING, ts=1428170136774, server=localhost,52251,1428170136570} 2015-04-04 10:55:36,810 INFO [AM.ZK.Worker-pool3-t15] master.RegionStates(1112): Transition {5bcaf4310f0272d29522a5c3ece41cf2 state=OPENING, ts=1428170136774, server=localhost,52251,1428170136570} to {5bcaf4310f0272d29522a5c3ece41cf2 state=OPEN, ts=1428170136810, server=localhost,52251,1428170136570} 2015-04-04 10:55:36,810 DEBUG [AM.ZK.Worker-pool3-t15] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 5bcaf4310f0272d29522a5c3ece41cf2 from localhost,52158,1428170104089; deleting unassigned node 2015-04-04 10:55:36,811 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:36,811 DEBUG [AM.ZK.Worker-pool3-t15] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 5bcaf4310f0272d29522a5c3ece41cf2 in expected state RS_ZK_REGION_OPENED 2015-04-04 10:55:36,811 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:36,812 DEBUG [AM.ZK.Worker-pool3-t15] master.AssignmentManager$4(1337): Znode TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. deleted, state: {5bcaf4310f0272d29522a5c3ece41cf2 state=OPEN, ts=1428170136810, server=localhost,52251,1428170136570} 2015-04-04 10:55:36,812 DEBUG [AM.ZK.Worker-pool3-t15] master.RegionStates(460): Onlined 5bcaf4310f0272d29522a5c3ece41cf2 on localhost,52251,1428170136570 {ENCODED => 5bcaf4310f0272d29522a5c3ece41cf2, NAME => 'TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:36,812 INFO [AM.ZK.Worker-pool3-t15] master.RegionStates(472): Offlined 5bcaf4310f0272d29522a5c3ece41cf2 from localhost,52161,1428170104473 2015-04-04 10:55:36,868 DEBUG [hconnection-0x5e481993-shared--pool14-t40] ipc.RpcClientImpl$Connection(343): Use SIMPLE authentication for service ClientService, sasl=false 2015-04-04 10:55:36,868 DEBUG [hconnection-0x5e481993-shared--pool14-t40] ipc.RpcClientImpl$Connection(708): Connecting to localhost/127.0.0.1:52251 2015-04-04 10:55:36,869 DEBUG [RpcServer.listener,port=52251] ipc.RpcServer$Listener(829): RpcServer.listener,port=52251: connection from 127.0.0.1:52261; # active connections: 2 2015-04-04 10:55:36,870 INFO [RpcServer.reader=2,bindAddress=localhost,port=52251] ipc.RpcServer$Connection(1661): Connection from 127.0.0.1 port: 52261 with version info: version: "1.1.0-SNAPSHOT" url: "git://apurtell-ltm2.internal.salesforce.com/Users/apurtell/src/hbase" revision: "5c19f9eac7d1f22147345f09aad2f04fec1ad67d" user: "apurtell" date: "Sat Apr 4 10:54:41 PDT 2015" src_checksum: "2a921de567e8e6d9e1c358f3276c1ec9" 2015-04-04 10:55:36,876 INFO [Thread-346] hbase.MiniHBaseCluster(245): Killing localhost,52251,1428170136570 2015-04-04 10:55:36,876 FATAL [Thread-346] regionserver.HRegionServer(1982): ABORTING region server localhost,52251,1428170136570: Simulated kill 2015-04-04 10:55:36,876 FATAL [Thread-346] regionserver.HRegionServer(1988): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy, org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver, org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2015-04-04 10:55:36,880 INFO [Thread-346] regionserver.HRegionServer(1992): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "HeapMemoryUsage" : { "committed" : 358612992, "init" : 134217728, "max" : 2609905664, "used" : 310331880 }, "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "NonHeapMemoryUsage" : { "committed" : 60227584, "init" : 24576000, "max" : 318767104, "used" : 60040688 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "localhost", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 6, "numActiveHandler" : 0, "TotalCallTime_num_ops" : 344, "TotalCallTime_min" : 0, "TotalCallTime_max" : 51, "TotalCallTime_mean" : 1.694767441860465, "TotalCallTime_median" : 1.0, "TotalCallTime_75th_percentile" : 1.0, "TotalCallTime_95th_percentile" : 6.0, "TotalCallTime_99th_percentile" : 34.500000000000114, "QueueCallTime_num_ops" : 344, "QueueCallTime_min" : 0, "QueueCallTime_max" : 10, "QueueCallTime_mean" : 0.12790697674418605, "QueueCallTime_median" : 0.0, "QueueCallTime_75th_percentile" : 0.0, "QueueCallTime_95th_percentile" : 1.0, "QueueCallTime_99th_percentile" : 1.0, "authenticationFailures" : 0, "authorizationFailures" : 0, "authenticationSuccesses" : 0, "authorizationSuccesses" : 23, "ProcessCallTime_num_ops" : 344, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 50, "ProcessCallTime_mean" : 1.566860465116279, "ProcessCallTime_median" : 1.0, "ProcessCallTime_75th_percentile" : 1.0, "ProcessCallTime_95th_percentile" : 6.0, "ProcessCallTime_99th_percentile" : 29.0, "sentBytes" : 83400, "receivedBytes" : 31976 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "localhost", "sink.appliedOps" : 0, "sink.appliedBatches" : 0, "sink.ageOfLastAppliedOp" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:49474", "tag.serverName" : "localhost,52161,1428170104473", "tag.clusterId" : "81c27d3e-db00-4d35-891c-9756f3546cb2", "tag.Context" : "regionserver", "tag.Hostname" : "localhost", "regionCount" : 2, "storeCount" : 2, "hlogFileCount" : 2, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 10360, "storeFileSize" : 0, "regionServerStartTime" : 1428170104473, "totalRequestCount" : 350, "readRequestCount" : 139, "writeRequestCount" : 25, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0, "percentFilesLocalSecondaryRegions" : 0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1042890544, "blockCacheCount" : 0, "blockCacheSize" : 1071760, "blockCacheHitCount" : 0, "blockCacheMissCount" : 0, "blockCacheEvictionCount" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "blockedRequestCount" : 0, "splitSuccessCounnt" : 0, "splitRequestCount" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0.0, "Append_median" : 0.0, "Append_75th_percentile" : 0.0, "Append_95th_percentile" : 0.0, "Append_99th_percentile" : 0.0, "Delete_num_ops" : 4, "Delete_min" : 1, "Delete_max" : 4, "Delete_mean" : 2.0, "Delete_median" : 1.5, "Delete_75th_percentile" : 3.5, "Delete_95th_percentile" : 4.0, "Delete_99th_percentile" : 4.0, "Mutate_num_ops" : 21, "Mutate_min" : 1, "Mutate_max" : 27, "Mutate_mean" : 3.0952380952380953, "Mutate_median" : 2.0, "Mutate_75th_percentile" : 2.0, "Mutate_95th_percentile" : 24.599999999999966, "Mutate_99th_percentile" : 27.0, "ScanNext_num_ops" : 100, "ScanNext_min" : 0, "ScanNext_max" : 1072, "ScanNext_mean" : 768.17, "ScanNext_median" : 1017.0, "ScanNext_75th_percentile" : 1042.0, "ScanNext_95th_percentile" : 1072.0, "ScanNext_99th_percentile" : 1072.0, "slowDeleteCount" : 0, "slowIncrementCount" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0.0, "FlushTime_median" : 0.0, "FlushTime_75th_percentile" : 0.0, "FlushTime_95th_percentile" : 0.0, "FlushTime_99th_percentile" : 0.0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 3, "Get_mean" : 1.0, "Get_median" : 0.5, "Get_75th_percentile" : 2.5, "Get_95th_percentile" : 3.0, "Get_99th_percentile" : 3.0, "Replay_num_ops" : 2, "Replay_min" : 3, "Replay_max" : 4, "Replay_mean" : 3.5, "Replay_median" : 3.5, "Replay_75th_percentile" : 4.0, "Replay_95th_percentile" : 4.0, "Replay_99th_percentile" : 4.0, "slowGetCount" : 0, "slowAppendCount" : 0, "slowPutCount" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0.0, "SplitTime_median" : 0.0, "SplitTime_75th_percentile" : 0.0, "SplitTime_95th_percentile" : 0.0, "SplitTime_99th_percentile" : 0.0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0.0, "Increment_median" : 0.0, "Increment_75th_percentile" : 0.0, "Increment_95th_percentile" : 0.0, "Increment_99th_percentile" : 0.0 } ] } 2015-04-04 10:55:36,882 ERROR [B.defaultRpcServer.handler=4,queue=0,port=52158] master.MasterRpcServices(328): Region server localhost,52251,1428170136570 reported a fatal error: ABORTING region server localhost,52251,1428170136570: Simulated kill 2015-04-04 10:55:36,882 INFO [Thread-346] regionserver.HRegionServer(1823): STOPPED: Simulated kill 2015-04-04 10:55:36,882 INFO [RS:3;localhost:52251] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-04-04 10:55:36,883 INFO [RS:3;localhost:52251] regionserver.HRegionServer(955): Stopping infoServer 2015-04-04 10:55:36,883 INFO [SplitLogWorker-localhost:52251] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-04-04 10:55:36,883 INFO [SplitLogWorker-localhost:52251] regionserver.SplitLogWorker(155): SplitLogWorker localhost,52251,1428170136570 exiting 2015-04-04 10:55:36,886 INFO [RS:3;localhost:52251] log.Slf4jLog(67): Stopped SelectChannelConnector@0.0.0.0:0 2015-04-04 10:55:36,888 INFO [RS:3;localhost:52251] regionserver.HeapMemoryManager(198): Stoping HeapMemoryTuner chore. 2015-04-04 10:55:36,888 INFO [RS:3;localhost:52251] snapshot.RegionServerSnapshotManager(134): Stopping RegionServerSnapshotManager abruptly. 2015-04-04 10:55:36,888 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-04-04 10:55:36,888 INFO [RS:3;localhost:52251] flush.RegionServerFlushTableProcedureManager(113): Stopping region server flush procedure manager abruptly. 2015-04-04 10:55:36,888 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-04-04 10:55:36,889 DEBUG [RS:3;localhost:52251] zookeeper.MetaTableLocator(619): Stopping MetaTableLocator 2015-04-04 10:55:36,889 INFO [RS:3;localhost:52251] client.ConnectionManager$HConnectionImplementation(1675): Closing zookeeper sessionid=0x14c859259d50017 2015-04-04 10:55:36,890 DEBUG [RS:3;localhost:52251] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:55:36,891 DEBUG [RpcServer.reader=7,bindAddress=localhost,port=52161] ipc.RpcServer$Listener(861): RpcServer.listener,port=52161: DISCONNECTING client 127.0.0.1:52260 because read count=-1. Number of active connections: 6 2015-04-04 10:55:36,992 DEBUG [RS:3;localhost:52251] wal.FSHLog(1148): Closing WAL writer in /user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/WALs/localhost,52251,1428170136570 2015-04-04 10:55:36,997 INFO [IPC Server handler 6 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741853_1029{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 928 2015-04-04 10:55:37,401 DEBUG [RS:3;localhost:52251] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:55:37,402 DEBUG [RpcServer.reader=5,bindAddress=localhost,port=52158] ipc.RpcServer$Listener(861): RpcServer.listener,port=52158: DISCONNECTING client 127.0.0.1:52255 because read count=-1. Number of active connections: 3 2015-04-04 10:55:37,505 INFO [RS:3;localhost:52251] regionserver.Leases(146): RS:3;localhost:52251 closing leases 2015-04-04 10:55:37,505 INFO [RS:3;localhost:52251] regionserver.Leases(149): RS:3;localhost:52251 closed leases 2015-04-04 10:55:37,506 INFO [RS:3;localhost:52251] ipc.RpcServer(2193): Stopping server on 52251 2015-04-04 10:55:37,506 INFO [RpcServer.listener,port=52251] ipc.RpcServer$Listener(765): RpcServer.listener,port=52251: stopping 2015-04-04 10:55:37,507 INFO [RpcServer.responder] ipc.RpcServer$Responder(1007): RpcServer.responder: stopped 2015-04-04 10:55:37,508 INFO [RpcServer.responder] ipc.RpcServer$Responder(910): RpcServer.responder: stopping 2015-04-04 10:55:37,510 DEBUG [Thread-346-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52251-0x14c859259d50016, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/localhost,52251,1428170136570 2015-04-04 10:55:37,510 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/localhost,52251,1428170136570 2015-04-04 10:55:37,510 INFO [main-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [localhost,52251,1428170136570] 2015-04-04 10:55:37,510 DEBUG [Thread-346-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52251-0x14c859259d50016, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-04-04 10:55:37,510 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/localhost,52251,1428170136570 2015-04-04 10:55:37,806 INFO [localhost,52251,1428170136570_ChoreService_1] hbase.ScheduledChore(179): Chore: localhost,52251,1428170136570-MemstoreFlusherChore was stopped 2015-04-04 10:55:37,812 INFO [RS:3;localhost:52251] regionserver.HRegionServer(1067): stopping server localhost,52251,1428170136570; zookeeper connection closed. 2015-04-04 10:55:37,812 INFO [RS:3;localhost:52251] regionserver.HRegionServer(1070): RS:3;localhost:52251 exiting 2015-04-04 10:55:37,806 INFO [regionserver/localhost/127.0.0.1:0.logRoller] regionserver.LogRoller(163): LogRoller exiting. 2015-04-04 10:55:37,806 INFO [regionserver/localhost/127.0.0.1:0.leaseChecker] regionserver.Leases(146): regionserver/localhost/127.0.0.1:0.leaseChecker closing leases 2015-04-04 10:55:37,813 INFO [regionserver/localhost/127.0.0.1:0.leaseChecker] regionserver.Leases(149): regionserver/localhost/127.0.0.1:0.leaseChecker closed leases 2015-04-04 10:55:37,813 DEBUG [main-EventThread] master.AssignmentManager(3327): based on AM, current region=hbase:meta,,1.1588230740 is on server=localhost,52161,1428170104473 server being checked: localhost,52251,1428170136570 2015-04-04 10:55:37,814 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@51f4c07] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(191): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@51f4c07 2015-04-04 10:55:37,814 DEBUG [main-EventThread] master.ServerManager(627): Added=localhost,52251,1428170136570 to dead servers, submitted shutdown handler to be executed meta=false 2015-04-04 10:55:37,814 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-04-04 10:55:37,815 DEBUG [main-EventThread] zookeeper.ZKUtil(490): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,52161,1428170104473 2015-04-04 10:55:37,816 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(121): /hbase/rs/localhost,52251,1428170136570 znode expired, triggering replicatorRemoved event 2015-04-04 10:55:37,816 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-04-04 10:55:37,816 DEBUG [main-EventThread] zookeeper.ZKUtil(490): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,52161,1428170104473 2015-04-04 10:55:37,817 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/localhost,52161,1428170104473 2015-04-04 10:55:37,817 DEBUG [main-EventThread] zookeeper.ZKUtil(490): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,52161,1428170104473 2015-04-04 10:55:37,826 INFO [MASTER_SERVER_OPERATIONS-localhost:52158-2] handler.ServerShutdownHandler(206): Mark regions in recovery for crashed server localhost,52251,1428170136570 before assignment; regions=[{ENCODED => 5bcaf4310f0272d29522a5c3ece41cf2, NAME => 'TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:55:37,827 DEBUG [MASTER_SERVER_OPERATIONS-localhost:52158-2] zookeeper.ZKUtil(749): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Unable to get data of znode /hbase/recovering-regions/5bcaf4310f0272d29522a5c3ece41cf2 because node does not exist (not an error) 2015-04-04 10:55:37,829 DEBUG [MASTER_SERVER_OPERATIONS-localhost:52158-2] coordination.ZKSplitLogManagerCoordination(650): Marked 5bcaf4310f0272d29522a5c3ece41cf2 as recovering from localhost,52251,1428170136570: /hbase/recovering-regions/5bcaf4310f0272d29522a5c3ece41cf2/localhost,52251,1428170136570 2015-04-04 10:55:37,829 DEBUG [MASTER_SERVER_OPERATIONS-localhost:52158-2] master.RegionStates(532): Adding to processed servers localhost,52251,1428170136570 2015-04-04 10:55:37,829 INFO [MASTER_SERVER_OPERATIONS-localhost:52158-2] master.RegionStates(1112): Transition {5bcaf4310f0272d29522a5c3ece41cf2 state=OPEN, ts=1428170136812, server=localhost,52251,1428170136570} to {5bcaf4310f0272d29522a5c3ece41cf2 state=OFFLINE, ts=1428170137829, server=localhost,52251,1428170136570} 2015-04-04 10:55:37,829 INFO [MASTER_SERVER_OPERATIONS-localhost:52158-2] handler.ServerShutdownHandler(238): Reassigning 1 region(s) that localhost,52251,1428170136570 was carrying (and 0 regions(s) that were opening on this server) 2015-04-04 10:55:37,829 INFO [MASTER_SERVER_OPERATIONS-localhost:52158-2] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,52161,1428170104473 2015-04-04 10:55:37,829 DEBUG [MASTER_SERVER_OPERATIONS-localhost:52158-2] zookeeper.ZKAssign(174): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Async create of unassigned node 5bcaf4310f0272d29522a5c3ece41cf2 with OFFLINE state 2015-04-04 10:55:37,830 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:37,830 DEBUG [main-EventThread] master.OfflineCallback(69): rs={5bcaf4310f0272d29522a5c3ece41cf2 state=OFFLINE, ts=1428170137829, server=localhost,52251,1428170136570}, server=localhost,52161,1428170104473 2015-04-04 10:55:37,831 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={5bcaf4310f0272d29522a5c3ece41cf2 state=OFFLINE, ts=1428170137829, server=localhost,52251,1428170136570}, server=localhost,52161,1428170104473 2015-04-04 10:55:37,835 DEBUG [MASTER_SERVER_OPERATIONS-localhost:52158-2] master.AssignmentManager(1658): localhost,52161,1428170104473 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:55:37,835 INFO [MASTER_SERVER_OPERATIONS-localhost:52158-2] master.RegionStates(1112): Transition {5bcaf4310f0272d29522a5c3ece41cf2 state=OFFLINE, ts=1428170137829, server=localhost,52251,1428170136570} to {5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_OPEN, ts=1428170137835, server=localhost,52161,1428170104473} 2015-04-04 10:55:37,836 INFO [PriorityRpcServer.handler=7,queue=1,port=52161] regionserver.RSRpcServices(1390): Open TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:37,843 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 5bcaf4310f0272d29522a5c3ece41cf2 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:37,843 DEBUG [MASTER_SERVER_OPERATIONS-localhost:52158-2] master.AssignmentManager(1788): Bulk assigning done for localhost,52161,1428170104473 2015-04-04 10:55:37,843 INFO [MASTER_SERVER_OPERATIONS-localhost:52158-2] master.AssignmentManager(3252): Waiting for 5bcaf4310f0272d29522a5c3ece41cf2 to leave regions-in-transition, timeOut=15000 ms. 2015-04-04 10:55:37,845 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:37,845 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 5bcaf4310f0272d29522a5c3ece41cf2 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:37,845 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(6078): Opening region: {ENCODED => 5bcaf4310f0272d29522a5c3ece41cf2, NAME => 'TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:37,845 DEBUG [AM.ZK.Worker-pool3-t1] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,52161,1428170104473, region=5bcaf4310f0272d29522a5c3ece41cf2, current_state={5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_OPEN, ts=1428170137835, server=localhost,52161,1428170104473} 2015-04-04 10:55:37,846 INFO [AM.ZK.Worker-pool3-t1] master.RegionStates(1112): Transition {5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_OPEN, ts=1428170137835, server=localhost,52161,1428170104473} to {5bcaf4310f0272d29522a5c3ece41cf2 state=OPENING, ts=1428170137846, server=localhost,52161,1428170104473} 2015-04-04 10:55:37,846 DEBUG [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:37,846 DEBUG [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:37,846 INFO [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:55:37,847 INFO [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:55:37,847 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testLegacyRecovery 5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:37,847 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(721): Instantiated TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:37,851 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:37,852 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:37,854 DEBUG [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/a 2015-04-04 10:55:37,855 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:37,856 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:37,857 DEBUG [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/b 2015-04-04 10:55:37,860 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:37,860 INFO [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:37,861 DEBUG [StoreOpener-5bcaf4310f0272d29522a5c3ece41cf2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/c 2015-04-04 10:55:37,864 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:37,867 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:55:37,873 DEBUG [RS_OPEN_REGION-localhost:52161-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits/40000005.seqid to file, newSeqId=40000005, maxSeqId=5 2015-04-04 10:55:37,874 INFO [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(847): Onlined 5bcaf4310f0272d29522a5c3ece41cf2; next sequenceid=40000005 2015-04-04 10:55:37,874 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(643): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Attempting to retransition opening state of node 5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:37,876 INFO [PostOpenDeployTasks:5bcaf4310f0272d29522a5c3ece41cf2] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:37,878 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/recovering-regions/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:37,879 DEBUG [PostOpenDeployTasks:5bcaf4310f0272d29522a5c3ece41cf2] regionserver.HRegionServer(3103): Update last flushed sequence id of region 5bcaf4310f0272d29522a5c3ece41cf2 for localhost,52251,1428170136570 2015-04-04 10:55:37,882 INFO [PostOpenDeployTasks:5bcaf4310f0272d29522a5c3ece41cf2] hbase.MetaTableAccessor(1350): Updated row TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. with server=localhost,52161,1428170104473 2015-04-04 10:55:37,883 DEBUG [PostOpenDeployTasks:5bcaf4310f0272d29522a5c3ece41cf2] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:37,883 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 5bcaf4310f0272d29522a5c3ece41cf2 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:37,884 INFO [Thread-346] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:37,886 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:37,886 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 5bcaf4310f0272d29522a5c3ece41cf2 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:37,887 DEBUG [RS_OPEN_REGION-localhost:52161-1] coordination.ZkOpenRegionCoordination(91): Transitioned 5bcaf4310f0272d29522a5c3ece41cf2 to OPENED in zk on localhost,52161,1428170104473 2015-04-04 10:55:37,887 DEBUG [RS_OPEN_REGION-localhost:52161-1] handler.OpenRegionHandler(167): Opened TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. on localhost,52161,1428170104473 2015-04-04 10:55:37,888 DEBUG [AM.ZK.Worker-pool3-t5] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,52161,1428170104473, region=5bcaf4310f0272d29522a5c3ece41cf2, current_state={5bcaf4310f0272d29522a5c3ece41cf2 state=OPENING, ts=1428170137846, server=localhost,52161,1428170104473} 2015-04-04 10:55:37,888 INFO [AM.ZK.Worker-pool3-t5] master.RegionStates(1112): Transition {5bcaf4310f0272d29522a5c3ece41cf2 state=OPENING, ts=1428170137846, server=localhost,52161,1428170104473} to {5bcaf4310f0272d29522a5c3ece41cf2 state=OPEN, ts=1428170137888, server=localhost,52161,1428170104473} 2015-04-04 10:55:37,888 DEBUG [AM.ZK.Worker-pool3-t5] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 5bcaf4310f0272d29522a5c3ece41cf2 from localhost,52158,1428170104089; deleting unassigned node 2015-04-04 10:55:37,890 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:37,890 DEBUG [AM.ZK.Worker-pool3-t5] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 5bcaf4310f0272d29522a5c3ece41cf2 in expected state RS_ZK_REGION_OPENED 2015-04-04 10:55:37,890 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:37,891 DEBUG [AM.ZK.Worker-pool3-t20] master.AssignmentManager$4(1337): Znode TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. deleted, state: {5bcaf4310f0272d29522a5c3ece41cf2 state=OPEN, ts=1428170137888, server=localhost,52161,1428170104473} 2015-04-04 10:55:37,891 DEBUG [AM.ZK.Worker-pool3-t20] master.RegionStates(460): Onlined 5bcaf4310f0272d29522a5c3ece41cf2 on localhost,52161,1428170104473 {ENCODED => 5bcaf4310f0272d29522a5c3ece41cf2, NAME => 'TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:37,892 INFO [AM.ZK.Worker-pool3-t20] master.RegionStates(472): Offlined 5bcaf4310f0272d29522a5c3ece41cf2 from localhost,52251,1428170136570 2015-04-04 10:55:37,893 DEBUG [MASTER_SERVER_OPERATIONS-localhost:52158-2] master.DeadServer(118): Finished processing localhost,52251,1428170136570 2015-04-04 10:55:37,894 INFO [Thread-346] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:37,894 INFO [Thread-346] coprocessor.TestRegionObserverInterface(683): All regions assigned 2015-04-04 10:55:37,895 DEBUG [M_LOG_REPLAY_OPS-localhost:52158-2] master.MasterFileSystem(330): Renamed region directory: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/WALs/localhost,52251,1428170136570-splitting 2015-04-04 10:55:37,895 INFO [M_LOG_REPLAY_OPS-localhost:52158-2] master.SplitLogManager(560): dead splitlog workers [localhost,52251,1428170136570] 2015-04-04 10:55:37,895 INFO [Thread-346] client.HBaseAdmin$9(979): Started disable of TestTable.testLegacyRecovery 2015-04-04 10:55:37,895 INFO [B.defaultRpcServer.handler=2,queue=0,port=52158] master.HMaster(1659): Client=apurtell/null disable TestTable.testLegacyRecovery 2015-04-04 10:55:37,897 DEBUG [M_LOG_REPLAY_OPS-localhost:52158-2] master.SplitLogManager(254): Scheduling batch of logs to split 2015-04-04 10:55:37,897 INFO [M_LOG_REPLAY_OPS-localhost:52158-2] master.SplitLogManager(256): started splitting 1 logs in [hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/WALs/localhost,52251,1428170136570-splitting] for [localhost,52251,1428170136570] 2015-04-04 10:55:37,897 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testLegacyRecovery/write-master:521580000000001 2015-04-04 10:55:37,898 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2015-04-04 10:55:37,898 DEBUG [main-EventThread] coordination.ZkSplitLogWorkerCoordination(107): tasks arrived or departed 2015-04-04 10:55:37,898 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination(432): put up splitlog task at znode /hbase/splitWAL/WALs%2Flocalhost%2C52251%2C1428170136570-splitting%2Flocalhost%252C52251%252C1428170136570.default.1428170136645 2015-04-04 10:55:37,899 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination(465): task not yet acquired /hbase/splitWAL/WALs%2Flocalhost%2C52251%2C1428170136570-splitting%2Flocalhost%252C52251%252C1428170136570.default.1428170136645 ver = 0 2015-04-04 10:55:37,900 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Flocalhost%2C52251%2C1428170136570-splitting%2Flocalhost%252C52251%252C1428170136570.default.1428170136645 2015-04-04 10:55:37,900 INFO [SplitLogWorker-localhost:52161] coordination.ZkSplitLogWorkerCoordination(263): worker localhost,52161,1428170104473 acquired task /hbase/splitWAL/WALs%2Flocalhost%2C52251%2C1428170136570-splitting%2Flocalhost%252C52251%252C1428170136570.default.1428170136645 2015-04-04 10:55:37,903 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(560): task /hbase/splitWAL/WALs%2Flocalhost%2C52251%2C1428170136570-splitting%2Flocalhost%252C52251%252C1428170136570.default.1428170136645 acquired by localhost,52161,1428170104473 2015-04-04 10:55:37,909 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(129): Attempting to disable table TestTable.testLegacyRecovery 2015-04-04 10:55:37,909 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testLegacyRecovery state from DISABLING to DISABLING 2015-04-04 10:55:37,911 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:55:37,912 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. (offlining), current state: {5bcaf4310f0272d29522a5c3ece41cf2 state=OPEN, ts=1428170137891, server=localhost,52161,1428170104473} 2015-04-04 10:55:37,913 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Creating unassigned node 5bcaf4310f0272d29522a5c3ece41cf2 in a CLOSING state 2015-04-04 10:55:37,914 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:37,916 INFO [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {5bcaf4310f0272d29522a5c3ece41cf2 state=OPEN, ts=1428170137891, server=localhost,52161,1428170104473} to {5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_CLOSE, ts=1428170137916, server=localhost,52161,1428170104473} 2015-04-04 10:55:37,916 INFO [PriorityRpcServer.handler=4,queue=0,port=52161] regionserver.RSRpcServices(1018): Close 5bcaf4310f0272d29522a5c3ece41cf2, moving to null 2015-04-04 10:55:37,917 DEBUG [RS_CLOSE_REGION-localhost:52161-1] handler.CloseRegionHandler(119): Processing close of TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:37,919 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,52161,1428170104473 for region TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:37,920 DEBUG [Thread-346] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in TestTable.testLegacyRecovery 2015-04-04 10:55:37,922 DEBUG [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(1350): Closing TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.: disabling compactions & flushes 2015-04-04 10:55:37,922 DEBUG [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:37,924 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.-1] regionserver.HStore(879): Closed a 2015-04-04 10:55:37,925 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.-1] regionserver.HStore(879): Closed b 2015-04-04 10:55:37,926 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.-1] regionserver.HStore(879): Closed c 2015-04-04 10:55:37,938 DEBUG [RS_CLOSE_REGION-localhost:52161-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits/40000006.seqid to file, newSeqId=40000006, maxSeqId=40000005 2015-04-04 10:55:37,939 INFO [RS_LOG_REPLAY_OPS-localhost:52161-0] wal.WALSplitter(289): Splitting wal: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/WALs/localhost,52251,1428170136570-splitting/localhost%2C52251%2C1428170136570.default.1428170136645, length=928 2015-04-04 10:55:37,939 INFO [RS_LOG_REPLAY_OPS-localhost:52161-0] wal.WALSplitter(290): DistributedLogReplay = true 2015-04-04 10:55:37,941 DEBUG [RS_CLOSE_REGION-localhost:52161-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:55:37,941 DEBUG [RS_CLOSE_REGION-localhost:52161-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:55:37,942 INFO [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(1471): Closed TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:37,942 DEBUG [RS_CLOSE_REGION-localhost:52161-1] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 5bcaf4310f0272d29522a5c3ece41cf2 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:37,943 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Flocalhost%2C52251%2C1428170136570-splitting%2Flocalhost%252C52251%252C1428170136570.default.1428170136645 2015-04-04 10:55:37,943 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Flocalhost%2C52251%2C1428170136570-splitting%2Flocalhost%252C52251%252C1428170136570.default.1428170136645 2015-04-04 10:55:37,943 INFO [RS_LOG_REPLAY_OPS-localhost:52161-0] util.FSHDFSUtils(173): Recovering lease on dfs file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/WALs/localhost,52251,1428170136570-splitting/localhost%2C52251%2C1428170136570.default.1428170136645 2015-04-04 10:55:37,945 INFO [RS_LOG_REPLAY_OPS-localhost:52161-0] util.FSHDFSUtils(260): recoverLease=true, attempt=0 on file=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/WALs/localhost,52251,1428170136570-splitting/localhost%2C52251%2C1428170136570.default.1428170136645 after 2ms 2015-04-04 10:55:37,945 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:37,945 DEBUG [RS_CLOSE_REGION-localhost:52161-1] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 5bcaf4310f0272d29522a5c3ece41cf2 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:37,946 DEBUG [RS_CLOSE_REGION-localhost:52161-1] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. on localhost,52161,1428170104473 2015-04-04 10:55:37,946 DEBUG [RS_CLOSE_REGION-localhost:52161-1] handler.CloseRegionHandler(164): Closed TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:37,947 DEBUG [AM.ZK.Worker-pool3-t4] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,52161,1428170104473, region=5bcaf4310f0272d29522a5c3ece41cf2, current_state={5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_CLOSE, ts=1428170137916, server=localhost,52161,1428170104473} 2015-04-04 10:55:37,947 DEBUG [AM.ZK.Worker-pool3-t4] handler.ClosedRegionHandler(93): Handling CLOSED event for 5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:37,947 DEBUG [AM.ZK.Worker-pool3-t4] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. 2015-04-04 10:55:37,949 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:37,949 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:37,949 DEBUG [AM.ZK.Worker-pool3-t4] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 5bcaf4310f0272d29522a5c3ece41cf2 in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:55:37,949 INFO [AM.ZK.Worker-pool3-t4] master.RegionStates(1112): Transition {5bcaf4310f0272d29522a5c3ece41cf2 state=PENDING_CLOSE, ts=1428170137916, server=localhost,52161,1428170104473} to {5bcaf4310f0272d29522a5c3ece41cf2 state=OFFLINE, ts=1428170137949, server=localhost,52161,1428170104473} 2015-04-04 10:55:37,950 INFO [AM.ZK.Worker-pool3-t4] master.RegionStates(600): Offlined 5bcaf4310f0272d29522a5c3ece41cf2 from localhost,52161,1428170104473 2015-04-04 10:55:37,956 DEBUG [RS_LOG_REPLAY_OPS-localhost:52161-0-Writer-0] wal.WALSplitter$WriterThread(1074): Writer thread Thread[RS_LOG_REPLAY_OPS-localhost:52161-0-Writer-0,5,main]: starting 2015-04-04 10:55:37,956 DEBUG [RS_LOG_REPLAY_OPS-localhost:52161-0-Writer-1] wal.WALSplitter$WriterThread(1074): Writer thread Thread[RS_LOG_REPLAY_OPS-localhost:52161-0-Writer-1,5,main]: starting 2015-04-04 10:55:37,958 DEBUG [RS_LOG_REPLAY_OPS-localhost:52161-0-Writer-2] wal.WALSplitter$WriterThread(1074): Writer thread Thread[RS_LOG_REPLAY_OPS-localhost:52161-0-Writer-2,5,main]: starting 2015-04-04 10:55:37,960 DEBUG [RS_LOG_REPLAY_OPS-localhost:52161-0] wal.WALSplitter(385): Finishing writing output logs and closing down. 2015-04-04 10:55:37,961 DEBUG [RS_LOG_REPLAY_OPS-localhost:52161-0] wal.WALSplitter$OutputSink(1205): Waiting for split writer threads to finish 2015-04-04 10:55:37,965 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testLegacyRecovery state from DISABLING to DISABLED 2015-04-04 10:55:37,967 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(191): Disabled table, TestTable.testLegacyRecovery, is done=true 2015-04-04 10:55:37,969 INFO [RS_LOG_REPLAY_OPS-localhost:52161-0-Writer-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1493): Creating writer path=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits/0000000000000000006.temp region=5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:37,969 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testLegacyRecovery/write-master:521580000000001 2015-04-04 10:55:37,971 INFO [RS_LOG_REPLAY_OPS-localhost:52161-0] wal.WALSplitter$OutputSink(1229): Split writers finished 2015-04-04 10:55:37,971 DEBUG [RS_LOG_REPLAY_OPS-localhost:52161-0] wal.WALSplitter$OutputSink(1205): Waiting for split writer threads to finish 2015-04-04 10:55:37,971 INFO [RS_LOG_REPLAY_OPS-localhost:52161-0] wal.WALSplitter$OutputSink(1229): Split writers finished 2015-04-04 10:55:37,972 DEBUG [RS_LOG_REPLAY_OPS-localhost:52161-0] wal.WALSplitter$LogRecoveredEditsOutputSink(1320): Submitting close of hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits/0000000000000000006.temp 2015-04-04 10:55:37,973 DEBUG [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink$2(1325): Closing hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits/0000000000000000006.temp 2015-04-04 10:55:37,981 INFO [IPC Server handler 5 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741854_1030{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 0 2015-04-04 10:55:37,982 INFO [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink$2(1333): Closed wap hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits/0000000000000000006.temp (wrote 2 edits in 10ms) 2015-04-04 10:55:37,985 INFO [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink$2(1364): Rename hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits/0000000000000000006.temp to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits/0000000000000000008 2015-04-04 10:55:37,986 INFO [RS_LOG_REPLAY_OPS-localhost:52161-0] wal.WALSplitter(407): Processed 2 edits across 1 regions; edits skipped=1; log file=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/WALs/localhost,52251,1428170136570-splitting/localhost%2C52251%2C1428170136570.default.1428170136645, length=928, corrupted=false, progress failed=false 2015-04-04 10:55:37,988 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Flocalhost%2C52251%2C1428170136570-splitting%2Flocalhost%252C52251%252C1428170136570.default.1428170136645 2015-04-04 10:55:37,988 INFO [RS_LOG_REPLAY_OPS-localhost:52161-0] coordination.ZkSplitLogWorkerCoordination(596): successfully transitioned task /hbase/splitWAL/WALs%2Flocalhost%2C52251%2C1428170136570-splitting%2Flocalhost%252C52251%252C1428170136570.default.1428170136645 to final state DONE localhost,52161,1428170104473 2015-04-04 10:55:37,988 INFO [RS_LOG_REPLAY_OPS-localhost:52161-0] handler.WALSplitterHandler(101): worker localhost,52161,1428170104473 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@686a5ea5 in 87ms 2015-04-04 10:55:37,988 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Flocalhost%2C52251%2C1428170136570-splitting%2Flocalhost%252C52251%252C1428170136570.default.1428170136645 2015-04-04 10:55:37,989 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(473): task /hbase/splitWAL/WALs%2Flocalhost%2C52251%2C1428170136570-splitting%2Flocalhost%252C52251%252C1428170136570.default.1428170136645 entered state: DONE localhost,52161,1428170104473 2015-04-04 10:55:37,995 INFO [main-EventThread] wal.WALSplitter(497): Archived processed log hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/WALs/localhost,52251,1428170136570-splitting/localhost%2C52251%2C1428170136570.default.1428170136645 to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/oldWALs/localhost%2C52251%2C1428170136570.default.1428170136645 2015-04-04 10:55:37,996 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(516): Done splitting /hbase/splitWAL/WALs%2Flocalhost%2C52251%2C1428170136570-splitting%2Flocalhost%252C52251%252C1428170136570.default.1428170136645 2015-04-04 10:55:37,997 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Flocalhost%2C52251%2C1428170136570-splitting%2Flocalhost%252C52251%252C1428170136570.default.1428170136645 2015-04-04 10:55:37,997 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2015-04-04 10:55:37,997 DEBUG [main-EventThread] coordination.ZkSplitLogWorkerCoordination(107): tasks arrived or departed 2015-04-04 10:55:37,997 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Flocalhost%2C52251%2C1428170136570-splitting%2Flocalhost%252C52251%252C1428170136570.default.1428170136645 2015-04-04 10:55:37,998 DEBUG [M_LOG_REPLAY_OPS-localhost:52158-2] coordination.ZKSplitLogManagerCoordination(324): Processing recovering [5bcaf4310f0272d29522a5c3ece41cf2] and servers [localhost,52251,1428170136570], isMetaRecovery=false 2015-04-04 10:55:37,998 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination$DeleteAsyncCallback(1078): deleted /hbase/splitWAL/WALs%2Flocalhost%2C52251%2C1428170136570-splitting%2Flocalhost%252C52251%252C1428170136570.default.1428170136645 2015-04-04 10:55:38,000 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/5bcaf4310f0272d29522a5c3ece41cf2/localhost,52251,1428170136570 2015-04-04 10:55:38,001 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:38,001 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:38,003 INFO [RS_OPEN_REGION-localhost:52161-2] handler.FinishRegionRecoveringHandler(52): /hbase/recovering-regions/5bcaf4310f0272d29522a5c3ece41cf2 deleted; 5bcaf4310f0272d29522a5c3ece41cf2 recovered. 2015-04-04 10:55:38,004 INFO [M_LOG_REPLAY_OPS-localhost:52158-2] master.SplitLogManager(315): finished splitting (more than or equal to) 928 bytes in 1 log files in [hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/WALs/localhost,52251,1428170136570-splitting] in 107ms 2015-04-04 10:55:38,004 DEBUG [M_LOG_REPLAY_OPS-localhost:52158-2] master.DeadServer(118): Finished processing localhost,52251,1428170136570 2015-04-04 10:55:38,004 INFO [M_LOG_REPLAY_OPS-localhost:52158-2] handler.LogReplayHandler(86): Finished processing shutdown of localhost,52251,1428170136570 2015-04-04 10:55:38,029 INFO [Thread-346] client.HBaseAdmin(1034): Disabled TestTable.testLegacyRecovery 2015-04-04 10:55:38,030 INFO [B.defaultRpcServer.handler=4,queue=0,port=52158] master.HMaster(1564): Client=apurtell/null delete TestTable.testLegacyRecovery 2015-04-04 10:55:38,032 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testLegacyRecovery/write-master:521580000000002 2015-04-04 10:55:38,037 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table TestTable.testLegacyRecovery 2015-04-04 10:55:38,042 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:55:38,044 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => 5bcaf4310f0272d29522a5c3ece41cf2, NAME => 'TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:55:38,047 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(186): Archiving region TestTable.testLegacyRecovery,,1428170135378.5bcaf4310f0272d29522a5c3ece41cf2. from FS 2015-04-04 10:55:38,047 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:38,052 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits] 2015-04-04 10:55:38,067 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits/0000000000000000008, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits/0000000000000000008 2015-04-04 10:55:38,071 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits/40000006.seqid, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2/recovered.edits/40000006.seqid 2015-04-04 10:55:38,071 INFO [IPC Server handler 8 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741852_1028 127.0.0.1:52151 2015-04-04 10:55:38,073 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testLegacyRecovery/5bcaf4310f0272d29522a5c3ece41cf2 2015-04-04 10:55:38,075 INFO [IPC Server handler 6 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741851_1027 127.0.0.1:52151 2015-04-04 10:55:38,076 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(196): Table 'TestTable.testLegacyRecovery' archived! 2015-04-04 10:55:38,076 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(116): Removing 'TestTable.testLegacyRecovery' descriptor. 2015-04-04 10:55:38,077 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(122): Removing 'TestTable.testLegacyRecovery' from region states. 2015-04-04 10:55:38,077 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(126): Marking 'TestTable.testLegacyRecovery' as deleted. 2015-04-04 10:55:38,084 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testLegacyRecovery/write-master:521580000000002 2015-04-04 10:55:38,151 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=52158] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = TestTable.testLegacyRecovery org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testLegacyRecovery at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:55:38,152 INFO [Thread-346] client.HBaseAdmin(739): Deleted TestTable.testLegacyRecovery 2015-04-04 10:55:38,191 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testLegacyRecovery Thread=403 (was 368) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t43 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: localhost:52251Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t69 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: M_LOG_REPLAY_OPS-localhost:52158-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t67 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: localhost,52251,1428170136570_ChoreService_1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t40 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-89293784_1 at /127.0.0.1:52247 [Waiting for operation #5] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t66 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t75 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS:3;localhost:52251-SendThread(localhost:49474) sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Potentially hanging thread: IPC Client (1382746753) connection to localhost/127.0.0.1:52251 from apurtell java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.waitForWork(RpcClientImpl.java:542) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:557) Potentially hanging thread: hconnection-0x6313aad-shared--pool9-t9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t70 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: LeaseRenewer:apurtell.hfs.3@localhost:52150 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t71 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS:3;localhost:52251-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t68 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: MASTER_SERVER_OPERATIONS-localhost:52158-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t79 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t77 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t38 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t73 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t44 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-89293784_1 at /127.0.0.1:52222 [Waiting for operation #28] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t74 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t78 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t72 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: IPC Client (1382746753) connection to localhost/127.0.0.1:52251 from apurtell java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.waitForWork(RpcClientImpl.java:542) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:557) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t41 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t42 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: IPC Client (1382746753) connection to localhost/127.0.0.1:52150 from apurtell.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:904) org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t76 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6313aad-shared--pool9-t8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-89293784_1 at /127.0.0.1:52248 [Waiting for operation #5] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS_OPEN_REGION-localhost:52251-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t39 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=439 (was 424) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=286 (was 285) - SystemLoadAverage LEAK? -, ProcessCount=236 (was 236), AvailableMemoryMB=17 (was 17), ConnectionCount=1 (was 1) 2015-04-04 10:55:38,224 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testHBase3583 Thread=403, OpenFileDescriptor=439, MaxFileDescriptor=10240, SystemLoadAverage=286, ProcessCount=236, AvailableMemoryMB=19, ConnectionCount=1 2015-04-04 10:55:38,227 INFO [B.defaultRpcServer.handler=3,queue=0,port=52158] master.HMaster(1318): Client=apurtell/null create 'testHBase3583', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:55:38,231 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testHBase3583/write-master:521580000000000 2015-04-04 10:55:38,237 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(188): Create table testHBase3583 2015-04-04 10:55:38,247 INFO [IPC Server handler 7 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741855_1031{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 0 2015-04-04 10:55:38,249 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3583/.tabledesc/.tableinfo.0000000001 2015-04-04 10:55:38,251 INFO [RegionOpenAndInitThread-testHBase3583-1] regionserver.HRegion(5885): creating HRegion testHBase3583 HTD == 'testHBase3583', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp Table name == testHBase3583 2015-04-04 10:55:38,260 INFO [IPC Server handler 7 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741856_1032{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 0 2015-04-04 10:55:38,261 DEBUG [RegionOpenAndInitThread-testHBase3583-1] regionserver.HRegion(721): Instantiated testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. 2015-04-04 10:55:38,261 DEBUG [RegionOpenAndInitThread-testHBase3583-1] regionserver.HRegion(1350): Closing testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a.: disabling compactions & flushes 2015-04-04 10:55:38,262 DEBUG [RegionOpenAndInitThread-testHBase3583-1] regionserver.HRegion(1377): Updates disabled for region testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. 2015-04-04 10:55:38,262 INFO [RegionOpenAndInitThread-testHBase3583-1] regionserver.HRegion(1471): Closed testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. 2015-04-04 10:55:38,266 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:55:38,267 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,52161,1428170104473 2015-04-04 10:55:38,267 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKAssign(174): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Async create of unassigned node 9e18bb3eb5b81a33518391f6b971222a with OFFLINE state 2015-04-04 10:55:38,268 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:38,268 DEBUG [main-EventThread] master.OfflineCallback(69): rs={9e18bb3eb5b81a33518391f6b971222a state=OFFLINE, ts=1428170138267, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:38,269 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={9e18bb3eb5b81a33518391f6b971222a state=OFFLINE, ts=1428170138267, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:38,273 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1658): localhost,52161,1428170104473 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:55:38,274 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.RegionStates(1112): Transition {9e18bb3eb5b81a33518391f6b971222a state=OFFLINE, ts=1428170138267, server=null} to {9e18bb3eb5b81a33518391f6b971222a state=PENDING_OPEN, ts=1428170138274, server=localhost,52161,1428170104473} 2015-04-04 10:55:38,275 INFO [PriorityRpcServer.handler=3,queue=1,port=52161] regionserver.RSRpcServices(1390): Open testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. 2015-04-04 10:55:38,281 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 9e18bb3eb5b81a33518391f6b971222a from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:38,281 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1788): Bulk assigning done for localhost,52161,1428170104473 2015-04-04 10:55:38,282 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table testHBase3583 state from ENABLING to ENABLED 2015-04-04 10:55:38,283 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 9e18bb3eb5b81a33518391f6b971222a from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:38,283 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/9e18bb3eb5b81a33518391f6b971222a 2015-04-04 10:55:38,284 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(6078): Opening region: {ENCODED => 9e18bb3eb5b81a33518391f6b971222a, NAME => 'testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:38,284 DEBUG [AM.ZK.Worker-pool3-t13] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,52161,1428170104473, region=9e18bb3eb5b81a33518391f6b971222a, current_state={9e18bb3eb5b81a33518391f6b971222a state=PENDING_OPEN, ts=1428170138274, server=localhost,52161,1428170104473} 2015-04-04 10:55:38,284 DEBUG [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:38,284 INFO [AM.ZK.Worker-pool3-t13] master.RegionStates(1112): Transition {9e18bb3eb5b81a33518391f6b971222a state=PENDING_OPEN, ts=1428170138274, server=localhost,52161,1428170104473} to {9e18bb3eb5b81a33518391f6b971222a state=OPENING, ts=1428170138284, server=localhost,52161,1428170104473} 2015-04-04 10:55:38,284 DEBUG [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:38,285 INFO [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:55:38,285 INFO [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:55:38,285 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testHBase3583 9e18bb3eb5b81a33518391f6b971222a 2015-04-04 10:55:38,285 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(721): Instantiated testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. 2015-04-04 10:55:38,289 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testHBase3583/write-master:521580000000000 2015-04-04 10:55:38,289 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(218): Table, testHBase3583, creation successful 2015-04-04 10:55:38,289 INFO [StoreOpener-9e18bb3eb5b81a33518391f6b971222a-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:38,289 INFO [StoreOpener-9e18bb3eb5b81a33518391f6b971222a-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:38,290 DEBUG [StoreOpener-9e18bb3eb5b81a33518391f6b971222a-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/a 2015-04-04 10:55:38,293 INFO [StoreOpener-9e18bb3eb5b81a33518391f6b971222a-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:38,293 INFO [StoreOpener-9e18bb3eb5b81a33518391f6b971222a-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:38,294 DEBUG [StoreOpener-9e18bb3eb5b81a33518391f6b971222a-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/b 2015-04-04 10:55:38,296 INFO [StoreOpener-9e18bb3eb5b81a33518391f6b971222a-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:38,297 INFO [StoreOpener-9e18bb3eb5b81a33518391f6b971222a-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:38,298 DEBUG [StoreOpener-9e18bb3eb5b81a33518391f6b971222a-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/c 2015-04-04 10:55:38,299 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a 2015-04-04 10:55:38,301 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:55:38,304 DEBUG [RS_OPEN_REGION-localhost:52161-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:55:38,305 INFO [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(847): Onlined 9e18bb3eb5b81a33518391f6b971222a; next sequenceid=2 2015-04-04 10:55:38,306 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(643): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Attempting to retransition opening state of node 9e18bb3eb5b81a33518391f6b971222a 2015-04-04 10:55:38,307 INFO [PostOpenDeployTasks:9e18bb3eb5b81a33518391f6b971222a] regionserver.HRegionServer(1849): Post open deploy tasks for testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. 2015-04-04 10:55:38,310 INFO [PostOpenDeployTasks:9e18bb3eb5b81a33518391f6b971222a] hbase.MetaTableAccessor(1350): Updated row testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. with server=localhost,52161,1428170104473 2015-04-04 10:55:38,310 DEBUG [PostOpenDeployTasks:9e18bb3eb5b81a33518391f6b971222a] regionserver.HRegionServer(1883): Finished post open deploy task for testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. 2015-04-04 10:55:38,311 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 9e18bb3eb5b81a33518391f6b971222a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:38,313 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/9e18bb3eb5b81a33518391f6b971222a 2015-04-04 10:55:38,313 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 9e18bb3eb5b81a33518391f6b971222a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:38,313 DEBUG [RS_OPEN_REGION-localhost:52161-0] coordination.ZkOpenRegionCoordination(91): Transitioned 9e18bb3eb5b81a33518391f6b971222a to OPENED in zk on localhost,52161,1428170104473 2015-04-04 10:55:38,313 DEBUG [RS_OPEN_REGION-localhost:52161-0] handler.OpenRegionHandler(167): Opened testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. on localhost,52161,1428170104473 2015-04-04 10:55:38,313 DEBUG [AM.ZK.Worker-pool3-t10] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,52161,1428170104473, region=9e18bb3eb5b81a33518391f6b971222a, current_state={9e18bb3eb5b81a33518391f6b971222a state=OPENING, ts=1428170138284, server=localhost,52161,1428170104473} 2015-04-04 10:55:38,313 INFO [AM.ZK.Worker-pool3-t10] master.RegionStates(1112): Transition {9e18bb3eb5b81a33518391f6b971222a state=OPENING, ts=1428170138284, server=localhost,52161,1428170104473} to {9e18bb3eb5b81a33518391f6b971222a state=OPEN, ts=1428170138313, server=localhost,52161,1428170104473} 2015-04-04 10:55:38,314 DEBUG [AM.ZK.Worker-pool3-t10] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 9e18bb3eb5b81a33518391f6b971222a from localhost,52158,1428170104089; deleting unassigned node 2015-04-04 10:55:38,315 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/9e18bb3eb5b81a33518391f6b971222a 2015-04-04 10:55:38,315 DEBUG [AM.ZK.Worker-pool3-t10] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 9e18bb3eb5b81a33518391f6b971222a in expected state RS_ZK_REGION_OPENED 2015-04-04 10:55:38,315 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:38,315 DEBUG [AM.ZK.Worker-pool3-t12] master.AssignmentManager$4(1337): Znode testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. deleted, state: {9e18bb3eb5b81a33518391f6b971222a state=OPEN, ts=1428170138313, server=localhost,52161,1428170104473} 2015-04-04 10:55:38,315 DEBUG [AM.ZK.Worker-pool3-t12] master.RegionStates(460): Onlined 9e18bb3eb5b81a33518391f6b971222a on localhost,52161,1428170104473 {ENCODED => 9e18bb3eb5b81a33518391f6b971222a, NAME => 'testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:38,351 INFO [Thread-400] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:38,356 INFO [Thread-400] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:38,357 INFO [Thread-400] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:38,364 INFO [Thread-400] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:38,381 INFO [Thread-400] client.HBaseAdmin$9(979): Started disable of testHBase3583 2015-04-04 10:55:38,382 INFO [B.defaultRpcServer.handler=0,queue=0,port=52158] master.HMaster(1659): Client=apurtell/null disable testHBase3583 2015-04-04 10:55:38,384 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testHBase3583/write-master:521580000000001 2015-04-04 10:55:38,389 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(129): Attempting to disable table testHBase3583 2015-04-04 10:55:38,390 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table testHBase3583 state from DISABLING to DISABLING 2015-04-04 10:55:38,391 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:55:38,391 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. (offlining), current state: {9e18bb3eb5b81a33518391f6b971222a state=OPEN, ts=1428170138315, server=localhost,52161,1428170104473} 2015-04-04 10:55:38,391 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Creating unassigned node 9e18bb3eb5b81a33518391f6b971222a in a CLOSING state 2015-04-04 10:55:38,392 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:38,393 INFO [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {9e18bb3eb5b81a33518391f6b971222a state=OPEN, ts=1428170138315, server=localhost,52161,1428170104473} to {9e18bb3eb5b81a33518391f6b971222a state=PENDING_CLOSE, ts=1428170138393, server=localhost,52161,1428170104473} 2015-04-04 10:55:38,394 INFO [PriorityRpcServer.handler=1,queue=1,port=52161] regionserver.RSRpcServices(1018): Close 9e18bb3eb5b81a33518391f6b971222a, moving to null 2015-04-04 10:55:38,394 DEBUG [RS_CLOSE_REGION-localhost:52161-2] handler.CloseRegionHandler(119): Processing close of testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. 2015-04-04 10:55:38,394 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,52161,1428170104473 for region testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. 2015-04-04 10:55:38,395 DEBUG [Thread-400] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in testHBase3583 2015-04-04 10:55:38,395 DEBUG [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegion(1350): Closing testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a.: disabling compactions & flushes 2015-04-04 10:55:38,395 DEBUG [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegion(1377): Updates disabled for region testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. 2015-04-04 10:55:38,396 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegion(2084): Started memstore flush for testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a., current region memstore size 160 B, and 3/3 column families' memstores are being flushed. 2015-04-04 10:55:38,407 INFO [IPC Server handler 7 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741857_1033{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 0 2015-04-04 10:55:38,408 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=6, memsize=160, hasBloomFilter=false, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/.tmp/0d259d1d38fc447d9bde4e9bbb83ec3e 2015-04-04 10:55:38,418 DEBUG [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/.tmp/0d259d1d38fc447d9bde4e9bbb83ec3e as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/a/0d259d1d38fc447d9bde4e9bbb83ec3e 2015-04-04 10:55:38,426 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/a/0d259d1d38fc447d9bde4e9bbb83ec3e, entries=1, sequenceid=6, filesize=4.6 K 2015-04-04 10:55:38,427 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegion(2363): Finished memstore flush of ~160 B/160, currentsize=0 B/0 for region testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. in 32ms, sequenceid=6, compaction requested=false 2015-04-04 10:55:38,429 INFO [StoreCloserThread-testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a.-1] regionserver.HStore(879): Closed a 2015-04-04 10:55:38,429 INFO [StoreCloserThread-testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a.-1] regionserver.HStore(879): Closed b 2015-04-04 10:55:38,429 INFO [StoreCloserThread-testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a.-1] regionserver.HStore(879): Closed c 2015-04-04 10:55:38,436 DEBUG [RS_CLOSE_REGION-localhost:52161-2] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/recovered.edits/9.seqid to file, newSeqId=9, maxSeqId=2 2015-04-04 10:55:38,437 DEBUG [RS_CLOSE_REGION-localhost:52161-2] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:55:38,438 DEBUG [RS_CLOSE_REGION-localhost:52161-2] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:55:38,438 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegion(1471): Closed testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. 2015-04-04 10:55:38,438 DEBUG [RS_CLOSE_REGION-localhost:52161-2] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 9e18bb3eb5b81a33518391f6b971222a from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:38,439 DEBUG [RS_CLOSE_REGION-localhost:52161-2] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 9e18bb3eb5b81a33518391f6b971222a from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:38,439 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/9e18bb3eb5b81a33518391f6b971222a 2015-04-04 10:55:38,440 DEBUG [RS_CLOSE_REGION-localhost:52161-2] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. on localhost,52161,1428170104473 2015-04-04 10:55:38,440 DEBUG [RS_CLOSE_REGION-localhost:52161-2] handler.CloseRegionHandler(164): Closed testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. 2015-04-04 10:55:38,440 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,52161,1428170104473, region=9e18bb3eb5b81a33518391f6b971222a, current_state={9e18bb3eb5b81a33518391f6b971222a state=PENDING_CLOSE, ts=1428170138393, server=localhost,52161,1428170104473} 2015-04-04 10:55:38,440 DEBUG [AM.ZK.Worker-pool3-t14] handler.ClosedRegionHandler(93): Handling CLOSED event for 9e18bb3eb5b81a33518391f6b971222a 2015-04-04 10:55:38,440 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. 2015-04-04 10:55:38,442 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/9e18bb3eb5b81a33518391f6b971222a 2015-04-04 10:55:38,442 DEBUG [AM.ZK.Worker-pool3-t14] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 9e18bb3eb5b81a33518391f6b971222a in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:55:38,442 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:38,442 INFO [AM.ZK.Worker-pool3-t14] master.RegionStates(1112): Transition {9e18bb3eb5b81a33518391f6b971222a state=PENDING_CLOSE, ts=1428170138393, server=localhost,52161,1428170104473} to {9e18bb3eb5b81a33518391f6b971222a state=OFFLINE, ts=1428170138442, server=localhost,52161,1428170104473} 2015-04-04 10:55:38,442 INFO [AM.ZK.Worker-pool3-t14] master.RegionStates(600): Offlined 9e18bb3eb5b81a33518391f6b971222a from localhost,52161,1428170104473 2015-04-04 10:55:38,494 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table testHBase3583 state from DISABLING to DISABLED 2015-04-04 10:55:38,496 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(191): Disabled table, testHBase3583, is done=true 2015-04-04 10:55:38,497 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testHBase3583/write-master:521580000000001 2015-04-04 10:55:38,502 INFO [Thread-400] client.HBaseAdmin(1034): Disabled testHBase3583 2015-04-04 10:55:38,503 INFO [B.defaultRpcServer.handler=2,queue=0,port=52158] master.HMaster(1564): Client=apurtell/null delete testHBase3583 2015-04-04 10:55:38,505 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testHBase3583/write-master:521580000000002 2015-04-04 10:55:38,509 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table testHBase3583 2015-04-04 10:55:38,512 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:55:38,515 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => 9e18bb3eb5b81a33518391f6b971222a, NAME => 'testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:55:38,517 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(186): Archiving region testHBase3583,,1428170138226.9e18bb3eb5b81a33518391f6b971222a. from FS 2015-04-04 10:55:38,517 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a 2015-04-04 10:55:38,522 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/recovered.edits] 2015-04-04 10:55:38,531 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/a/0d259d1d38fc447d9bde4e9bbb83ec3e, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/a/0d259d1d38fc447d9bde4e9bbb83ec3e 2015-04-04 10:55:38,543 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/recovered.edits/9.seqid, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a/recovered.edits/9.seqid 2015-04-04 10:55:38,544 INFO [IPC Server handler 3 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741856_1032 127.0.0.1:52151 2015-04-04 10:55:38,545 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3583/9e18bb3eb5b81a33518391f6b971222a 2015-04-04 10:55:38,546 INFO [IPC Server handler 1 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741855_1031 127.0.0.1:52151 2015-04-04 10:55:38,547 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(196): Table 'testHBase3583' archived! 2015-04-04 10:55:38,547 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(116): Removing 'testHBase3583' descriptor. 2015-04-04 10:55:38,548 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(122): Removing 'testHBase3583' from region states. 2015-04-04 10:55:38,548 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(126): Marking 'testHBase3583' as deleted. 2015-04-04 10:55:38,553 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testHBase3583/write-master:521580000000002 2015-04-04 10:55:38,621 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=52158] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = testHBase3583 org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3583 at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:55:38,621 INFO [Thread-400] client.HBaseAdmin(739): Deleted testHBase3583 2015-04-04 10:55:38,662 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testHBase3583 Thread=423 (was 403) Potentially hanging thread: hconnection-0x74f1d259-metaLookup-shared--pool77-t1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t46 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t81 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52247 [Waiting for operation #7] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t90 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t49 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6313aad-shared--pool9-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t80 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t48 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t91 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t47 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t87 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t50 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t82 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t88 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t89 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52222 [Waiting for operation #30] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t83 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t84 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t85 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t45 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-89293784_1 at /127.0.0.1:52248 [Waiting for operation #7] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t86 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=439 (was 439), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=286 (was 286), ProcessCount=236 (was 236), AvailableMemoryMB=15 (was 19), ConnectionCount=1 (was 1) 2015-04-04 10:55:38,694 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testHBase3758 Thread=423, OpenFileDescriptor=439, MaxFileDescriptor=10240, SystemLoadAverage=286, ProcessCount=236, AvailableMemoryMB=17, ConnectionCount=1 2015-04-04 10:55:38,696 INFO [B.defaultRpcServer.handler=1,queue=0,port=52158] master.HMaster(1318): Client=apurtell/null create 'testHBase3758', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:55:38,711 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testHBase3758/write-master:521580000000000 2015-04-04 10:55:38,718 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(188): Create table testHBase3758 2015-04-04 10:55:38,731 INFO [IPC Server handler 9 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741858_1034{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 777 2015-04-04 10:55:39,141 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3758/.tabledesc/.tableinfo.0000000001 2015-04-04 10:55:39,143 INFO [RegionOpenAndInitThread-testHBase3758-1] regionserver.HRegion(5885): creating HRegion testHBase3758 HTD == 'testHBase3758', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp Table name == testHBase3758 2015-04-04 10:55:39,153 INFO [IPC Server handler 9 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741859_1035{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 0 2015-04-04 10:55:39,154 DEBUG [RegionOpenAndInitThread-testHBase3758-1] regionserver.HRegion(721): Instantiated testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. 2015-04-04 10:55:39,154 DEBUG [RegionOpenAndInitThread-testHBase3758-1] regionserver.HRegion(1350): Closing testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929.: disabling compactions & flushes 2015-04-04 10:55:39,154 DEBUG [RegionOpenAndInitThread-testHBase3758-1] regionserver.HRegion(1377): Updates disabled for region testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. 2015-04-04 10:55:39,154 INFO [RegionOpenAndInitThread-testHBase3758-1] regionserver.HRegion(1471): Closed testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. 2015-04-04 10:55:39,158 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:55:39,159 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,52161,1428170104473 2015-04-04 10:55:39,159 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKAssign(174): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Async create of unassigned node 1d68d42c4bc9ed27d2bddec0be0d5929 with OFFLINE state 2015-04-04 10:55:39,160 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:39,160 DEBUG [main-EventThread] master.OfflineCallback(69): rs={1d68d42c4bc9ed27d2bddec0be0d5929 state=OFFLINE, ts=1428170139159, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:39,161 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={1d68d42c4bc9ed27d2bddec0be0d5929 state=OFFLINE, ts=1428170139159, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:39,165 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1658): localhost,52161,1428170104473 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:55:39,165 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.RegionStates(1112): Transition {1d68d42c4bc9ed27d2bddec0be0d5929 state=OFFLINE, ts=1428170139159, server=null} to {1d68d42c4bc9ed27d2bddec0be0d5929 state=PENDING_OPEN, ts=1428170139165, server=localhost,52161,1428170104473} 2015-04-04 10:55:39,166 INFO [PriorityRpcServer.handler=4,queue=0,port=52161] regionserver.RSRpcServices(1390): Open testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. 2015-04-04 10:55:39,171 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 1d68d42c4bc9ed27d2bddec0be0d5929 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:39,171 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1788): Bulk assigning done for localhost,52161,1428170104473 2015-04-04 10:55:39,171 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table testHBase3758 state from ENABLING to ENABLED 2015-04-04 10:55:39,173 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1d68d42c4bc9ed27d2bddec0be0d5929 2015-04-04 10:55:39,173 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 1d68d42c4bc9ed27d2bddec0be0d5929 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:39,173 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(6078): Opening region: {ENCODED => 1d68d42c4bc9ed27d2bddec0be0d5929, NAME => 'testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:39,173 DEBUG [AM.ZK.Worker-pool3-t2] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,52161,1428170104473, region=1d68d42c4bc9ed27d2bddec0be0d5929, current_state={1d68d42c4bc9ed27d2bddec0be0d5929 state=PENDING_OPEN, ts=1428170139165, server=localhost,52161,1428170104473} 2015-04-04 10:55:39,173 INFO [AM.ZK.Worker-pool3-t2] master.RegionStates(1112): Transition {1d68d42c4bc9ed27d2bddec0be0d5929 state=PENDING_OPEN, ts=1428170139165, server=localhost,52161,1428170104473} to {1d68d42c4bc9ed27d2bddec0be0d5929 state=OPENING, ts=1428170139173, server=localhost,52161,1428170104473} 2015-04-04 10:55:39,174 DEBUG [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:39,174 DEBUG [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:39,174 INFO [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:55:39,174 INFO [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:55:39,174 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testHBase3758 1d68d42c4bc9ed27d2bddec0be0d5929 2015-04-04 10:55:39,174 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(721): Instantiated testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. 2015-04-04 10:55:39,177 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testHBase3758/write-master:521580000000000 2015-04-04 10:55:39,177 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(218): Table, testHBase3758, creation successful 2015-04-04 10:55:39,178 INFO [StoreOpener-1d68d42c4bc9ed27d2bddec0be0d5929-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:39,178 INFO [StoreOpener-1d68d42c4bc9ed27d2bddec0be0d5929-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:39,180 DEBUG [StoreOpener-1d68d42c4bc9ed27d2bddec0be0d5929-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/a 2015-04-04 10:55:39,182 INFO [StoreOpener-1d68d42c4bc9ed27d2bddec0be0d5929-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:39,182 INFO [StoreOpener-1d68d42c4bc9ed27d2bddec0be0d5929-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:39,183 DEBUG [StoreOpener-1d68d42c4bc9ed27d2bddec0be0d5929-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/b 2015-04-04 10:55:39,185 INFO [StoreOpener-1d68d42c4bc9ed27d2bddec0be0d5929-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:39,185 INFO [StoreOpener-1d68d42c4bc9ed27d2bddec0be0d5929-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:39,187 DEBUG [StoreOpener-1d68d42c4bc9ed27d2bddec0be0d5929-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/c 2015-04-04 10:55:39,188 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929 2015-04-04 10:55:39,191 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:55:39,195 DEBUG [RS_OPEN_REGION-localhost:52161-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:55:39,195 INFO [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(847): Onlined 1d68d42c4bc9ed27d2bddec0be0d5929; next sequenceid=2 2015-04-04 10:55:39,196 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(643): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Attempting to retransition opening state of node 1d68d42c4bc9ed27d2bddec0be0d5929 2015-04-04 10:55:39,197 INFO [PostOpenDeployTasks:1d68d42c4bc9ed27d2bddec0be0d5929] regionserver.HRegionServer(1849): Post open deploy tasks for testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. 2015-04-04 10:55:39,200 INFO [PostOpenDeployTasks:1d68d42c4bc9ed27d2bddec0be0d5929] hbase.MetaTableAccessor(1350): Updated row testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. with server=localhost,52161,1428170104473 2015-04-04 10:55:39,200 DEBUG [PostOpenDeployTasks:1d68d42c4bc9ed27d2bddec0be0d5929] regionserver.HRegionServer(1883): Finished post open deploy task for testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. 2015-04-04 10:55:39,201 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 1d68d42c4bc9ed27d2bddec0be0d5929 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:39,202 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1d68d42c4bc9ed27d2bddec0be0d5929 2015-04-04 10:55:39,202 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 1d68d42c4bc9ed27d2bddec0be0d5929 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:39,203 DEBUG [RS_OPEN_REGION-localhost:52161-1] coordination.ZkOpenRegionCoordination(91): Transitioned 1d68d42c4bc9ed27d2bddec0be0d5929 to OPENED in zk on localhost,52161,1428170104473 2015-04-04 10:55:39,203 DEBUG [RS_OPEN_REGION-localhost:52161-1] handler.OpenRegionHandler(167): Opened testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. on localhost,52161,1428170104473 2015-04-04 10:55:39,204 DEBUG [AM.ZK.Worker-pool3-t3] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,52161,1428170104473, region=1d68d42c4bc9ed27d2bddec0be0d5929, current_state={1d68d42c4bc9ed27d2bddec0be0d5929 state=OPENING, ts=1428170139173, server=localhost,52161,1428170104473} 2015-04-04 10:55:39,204 INFO [AM.ZK.Worker-pool3-t3] master.RegionStates(1112): Transition {1d68d42c4bc9ed27d2bddec0be0d5929 state=OPENING, ts=1428170139173, server=localhost,52161,1428170104473} to {1d68d42c4bc9ed27d2bddec0be0d5929 state=OPEN, ts=1428170139204, server=localhost,52161,1428170104473} 2015-04-04 10:55:39,204 DEBUG [AM.ZK.Worker-pool3-t3] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 1d68d42c4bc9ed27d2bddec0be0d5929 from localhost,52158,1428170104089; deleting unassigned node 2015-04-04 10:55:39,205 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/1d68d42c4bc9ed27d2bddec0be0d5929 2015-04-04 10:55:39,205 DEBUG [AM.ZK.Worker-pool3-t3] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 1d68d42c4bc9ed27d2bddec0be0d5929 in expected state RS_ZK_REGION_OPENED 2015-04-04 10:55:39,205 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:39,206 DEBUG [AM.ZK.Worker-pool3-t3] master.AssignmentManager$4(1337): Znode testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. deleted, state: {1d68d42c4bc9ed27d2bddec0be0d5929 state=OPEN, ts=1428170139204, server=localhost,52161,1428170104473} 2015-04-04 10:55:39,206 DEBUG [AM.ZK.Worker-pool3-t3] master.RegionStates(460): Onlined 1d68d42c4bc9ed27d2bddec0be0d5929 on localhost,52161,1428170104473 {ENCODED => 1d68d42c4bc9ed27d2bddec0be0d5929, NAME => 'testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:39,356 INFO [Thread-415] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:39,361 INFO [Thread-415] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:39,375 INFO [Thread-415] client.HBaseAdmin$9(979): Started disable of testHBase3758 2015-04-04 10:55:39,376 INFO [B.defaultRpcServer.handler=4,queue=0,port=52158] master.HMaster(1659): Client=apurtell/null disable testHBase3758 2015-04-04 10:55:39,385 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testHBase3758/write-master:521580000000001 2015-04-04 10:55:39,392 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(129): Attempting to disable table testHBase3758 2015-04-04 10:55:39,392 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table testHBase3758 state from DISABLING to DISABLING 2015-04-04 10:55:39,394 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:55:39,395 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. (offlining), current state: {1d68d42c4bc9ed27d2bddec0be0d5929 state=OPEN, ts=1428170139206, server=localhost,52161,1428170104473} 2015-04-04 10:55:39,395 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Creating unassigned node 1d68d42c4bc9ed27d2bddec0be0d5929 in a CLOSING state 2015-04-04 10:55:39,398 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:39,399 INFO [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {1d68d42c4bc9ed27d2bddec0be0d5929 state=OPEN, ts=1428170139206, server=localhost,52161,1428170104473} to {1d68d42c4bc9ed27d2bddec0be0d5929 state=PENDING_CLOSE, ts=1428170139399, server=localhost,52161,1428170104473} 2015-04-04 10:55:39,400 DEBUG [Thread-415] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in testHBase3758 2015-04-04 10:55:39,400 INFO [PriorityRpcServer.handler=1,queue=1,port=52161] regionserver.RSRpcServices(1018): Close 1d68d42c4bc9ed27d2bddec0be0d5929, moving to null 2015-04-04 10:55:39,401 DEBUG [RS_CLOSE_REGION-localhost:52161-0] handler.CloseRegionHandler(119): Processing close of testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. 2015-04-04 10:55:39,401 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,52161,1428170104473 for region testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. 2015-04-04 10:55:39,402 DEBUG [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(1350): Closing testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929.: disabling compactions & flushes 2015-04-04 10:55:39,402 DEBUG [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(1377): Updates disabled for region testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. 2015-04-04 10:55:39,402 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(2084): Started memstore flush for testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929., current region memstore size 640 B, and 3/3 column families' memstores are being flushed. 2015-04-04 10:55:39,420 INFO [IPC Server handler 1 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741860_1036{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 4911 2015-04-04 10:55:39,510 DEBUG [Thread-415] client.HBaseAdmin(1019): Sleeping= 200ms, waiting for all regions to be disabled in testHBase3758 2015-04-04 10:55:39,716 DEBUG [Thread-415] client.HBaseAdmin(1019): Sleeping= 300ms, waiting for all regions to be disabled in testHBase3758 2015-04-04 10:55:39,826 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/.tmp/c32c11cfb5f84b68910cc206b4c9744b 2015-04-04 10:55:39,837 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c32c11cfb5f84b68910cc206b4c9744b 2015-04-04 10:55:39,846 INFO [IPC Server handler 7 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741861_1037{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 0 2015-04-04 10:55:39,847 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=160, hasBloomFilter=false, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/.tmp/e3c61b79e0f7495b91ff932bb2295641 2015-04-04 10:55:39,855 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e3c61b79e0f7495b91ff932bb2295641 2015-04-04 10:55:39,864 INFO [IPC Server handler 8 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741862_1038{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 0 2015-04-04 10:55:39,865 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=160, hasBloomFilter=false, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/.tmp/5509c52ca81e4b40b21c6a21fc094d83 2015-04-04 10:55:39,875 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5509c52ca81e4b40b21c6a21fc094d83 2015-04-04 10:55:39,877 DEBUG [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/.tmp/c32c11cfb5f84b68910cc206b4c9744b as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/a/c32c11cfb5f84b68910cc206b4c9744b 2015-04-04 10:55:39,888 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c32c11cfb5f84b68910cc206b4c9744b 2015-04-04 10:55:39,888 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/a/c32c11cfb5f84b68910cc206b4c9744b, entries=1, sequenceid=8, filesize=4.8 K 2015-04-04 10:55:39,892 DEBUG [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/.tmp/e3c61b79e0f7495b91ff932bb2295641 as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/b/e3c61b79e0f7495b91ff932bb2295641 2015-04-04 10:55:39,901 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e3c61b79e0f7495b91ff932bb2295641 2015-04-04 10:55:39,901 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/b/e3c61b79e0f7495b91ff932bb2295641, entries=1, sequenceid=8, filesize=4.8 K 2015-04-04 10:55:39,905 DEBUG [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/.tmp/5509c52ca81e4b40b21c6a21fc094d83 as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/c/5509c52ca81e4b40b21c6a21fc094d83 2015-04-04 10:55:39,915 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5509c52ca81e4b40b21c6a21fc094d83 2015-04-04 10:55:39,916 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/c/5509c52ca81e4b40b21c6a21fc094d83, entries=1, sequenceid=8, filesize=4.8 K 2015-04-04 10:55:39,917 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(2363): Finished memstore flush of ~640 B/640, currentsize=0 B/0 for region testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. in 515ms, sequenceid=8, compaction requested=false 2015-04-04 10:55:39,919 INFO [StoreCloserThread-testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929.-1] regionserver.HStore(879): Closed a 2015-04-04 10:55:39,920 INFO [StoreCloserThread-testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929.-1] regionserver.HStore(879): Closed b 2015-04-04 10:55:39,922 INFO [StoreCloserThread-testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929.-1] regionserver.HStore(879): Closed c 2015-04-04 10:55:39,930 DEBUG [RS_CLOSE_REGION-localhost:52161-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/recovered.edits/11.seqid to file, newSeqId=11, maxSeqId=2 2015-04-04 10:55:39,931 DEBUG [RS_CLOSE_REGION-localhost:52161-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:55:39,931 DEBUG [RS_CLOSE_REGION-localhost:52161-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:55:39,931 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(1471): Closed testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. 2015-04-04 10:55:39,931 DEBUG [RS_CLOSE_REGION-localhost:52161-0] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 1d68d42c4bc9ed27d2bddec0be0d5929 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:39,933 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1d68d42c4bc9ed27d2bddec0be0d5929 2015-04-04 10:55:39,933 DEBUG [RS_CLOSE_REGION-localhost:52161-0] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 1d68d42c4bc9ed27d2bddec0be0d5929 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:39,933 DEBUG [RS_CLOSE_REGION-localhost:52161-0] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. on localhost,52161,1428170104473 2015-04-04 10:55:39,933 DEBUG [RS_CLOSE_REGION-localhost:52161-0] handler.CloseRegionHandler(164): Closed testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. 2015-04-04 10:55:39,933 DEBUG [AM.ZK.Worker-pool3-t5] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,52161,1428170104473, region=1d68d42c4bc9ed27d2bddec0be0d5929, current_state={1d68d42c4bc9ed27d2bddec0be0d5929 state=PENDING_CLOSE, ts=1428170139399, server=localhost,52161,1428170104473} 2015-04-04 10:55:39,934 DEBUG [AM.ZK.Worker-pool3-t5] handler.ClosedRegionHandler(93): Handling CLOSED event for 1d68d42c4bc9ed27d2bddec0be0d5929 2015-04-04 10:55:39,934 DEBUG [AM.ZK.Worker-pool3-t5] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. 2015-04-04 10:55:39,935 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/1d68d42c4bc9ed27d2bddec0be0d5929 2015-04-04 10:55:39,935 DEBUG [AM.ZK.Worker-pool3-t5] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 1d68d42c4bc9ed27d2bddec0be0d5929 in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:55:39,935 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:39,935 INFO [AM.ZK.Worker-pool3-t5] master.RegionStates(1112): Transition {1d68d42c4bc9ed27d2bddec0be0d5929 state=PENDING_CLOSE, ts=1428170139399, server=localhost,52161,1428170104473} to {1d68d42c4bc9ed27d2bddec0be0d5929 state=OFFLINE, ts=1428170139935, server=localhost,52161,1428170104473} 2015-04-04 10:55:39,935 INFO [AM.ZK.Worker-pool3-t5] master.RegionStates(600): Offlined 1d68d42c4bc9ed27d2bddec0be0d5929 from localhost,52161,1428170104473 2015-04-04 10:55:39,978 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table testHBase3758 state from DISABLING to DISABLED 2015-04-04 10:55:39,980 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(191): Disabled table, testHBase3758, is done=true 2015-04-04 10:55:39,981 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testHBase3758/write-master:521580000000001 2015-04-04 10:55:40,024 INFO [Thread-415] client.HBaseAdmin(1034): Disabled testHBase3758 2015-04-04 10:55:40,025 INFO [B.defaultRpcServer.handler=2,queue=0,port=52158] master.HMaster(1564): Client=apurtell/null delete testHBase3758 2015-04-04 10:55:40,027 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testHBase3758/write-master:521580000000002 2015-04-04 10:55:40,031 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table testHBase3758 2015-04-04 10:55:40,035 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:55:40,037 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => 1d68d42c4bc9ed27d2bddec0be0d5929, NAME => 'testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:55:40,040 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(186): Archiving region testHBase3758,,1428170138696.1d68d42c4bc9ed27d2bddec0be0d5929. from FS 2015-04-04 10:55:40,040 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929 2015-04-04 10:55:40,045 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/recovered.edits] 2015-04-04 10:55:40,052 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/a/c32c11cfb5f84b68910cc206b4c9744b, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/a/c32c11cfb5f84b68910cc206b4c9744b 2015-04-04 10:55:40,059 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/b/e3c61b79e0f7495b91ff932bb2295641, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/b/e3c61b79e0f7495b91ff932bb2295641 2015-04-04 10:55:40,066 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/c/5509c52ca81e4b40b21c6a21fc094d83, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/c/5509c52ca81e4b40b21c6a21fc094d83 2015-04-04 10:55:40,073 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/recovered.edits/11.seqid, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929/recovered.edits/11.seqid 2015-04-04 10:55:40,074 INFO [IPC Server handler 3 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741859_1035 127.0.0.1:52151 2015-04-04 10:55:40,074 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/testHBase3758/1d68d42c4bc9ed27d2bddec0be0d5929 2015-04-04 10:55:40,075 INFO [IPC Server handler 9 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741858_1034 127.0.0.1:52151 2015-04-04 10:55:40,075 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(196): Table 'testHBase3758' archived! 2015-04-04 10:55:40,075 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(116): Removing 'testHBase3758' descriptor. 2015-04-04 10:55:40,076 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(122): Removing 'testHBase3758' from region states. 2015-04-04 10:55:40,076 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(126): Marking 'testHBase3758' as deleted. 2015-04-04 10:55:40,082 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testHBase3758/write-master:521580000000002 2015-04-04 10:55:40,144 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=52158] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = testHBase3758 org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/testHBase3758 at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:55:40,145 INFO [Thread-415] client.HBaseAdmin(739): Deleted testHBase3758 2015-04-04 10:55:40,179 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testHBase3758 Thread=447 (was 423) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t92 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t53 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t97 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6313aad-shared--pool9-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t55 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t51 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t94 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t102 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52247 [Waiting for operation #11] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t60 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x74f1d259-metaLookup-shared--pool77-t2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t103 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t101 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t95 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t52 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t98 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t59 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t58 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t93 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t57 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t56 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t99 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t100 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t96 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52222 [Waiting for operation #34] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t54 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52248 [Waiting for operation #13] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=439 (was 439), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=286 (was 286), ProcessCount=236 (was 236), AvailableMemoryMB=16 (was 17), ConnectionCount=1 (was 1) 2015-04-04 10:55:40,213 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testCompactionOverride Thread=447, OpenFileDescriptor=439, MaxFileDescriptor=10240, SystemLoadAverage=286, ProcessCount=236, AvailableMemoryMB=18, ConnectionCount=1 2015-04-04 10:55:40,228 INFO [B.defaultRpcServer.handler=1,queue=0,port=52158] master.HMaster(1318): Client=apurtell/null create 'TestCompactionOverride', {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor|1073741823|'}, {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:55:40,231 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestCompactionOverride/write-master:521580000000000 2015-04-04 10:55:40,238 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(188): Create table TestCompactionOverride 2015-04-04 10:55:40,247 INFO [IPC Server handler 5 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741863_1039{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 0 2015-04-04 10:55:40,250 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestCompactionOverride/.tabledesc/.tableinfo.0000000001 2015-04-04 10:55:40,251 INFO [RegionOpenAndInitThread-TestCompactionOverride-1] regionserver.HRegion(5885): creating HRegion TestCompactionOverride HTD == 'TestCompactionOverride', {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor|1073741823|'}, {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp Table name == TestCompactionOverride 2015-04-04 10:55:40,262 INFO [IPC Server handler 5 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741864_1040{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 57 2015-04-04 10:55:40,667 DEBUG [RegionOpenAndInitThread-TestCompactionOverride-1] regionserver.HRegion(721): Instantiated TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. 2015-04-04 10:55:40,668 DEBUG [RegionOpenAndInitThread-TestCompactionOverride-1] regionserver.HRegion(1350): Closing TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9.: disabling compactions & flushes 2015-04-04 10:55:40,668 DEBUG [RegionOpenAndInitThread-TestCompactionOverride-1] regionserver.HRegion(1377): Updates disabled for region TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. 2015-04-04 10:55:40,668 INFO [RegionOpenAndInitThread-TestCompactionOverride-1] regionserver.HRegion(1471): Closed TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. 2015-04-04 10:55:40,673 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:55:40,674 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,52161,1428170104473 2015-04-04 10:55:40,674 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKAssign(174): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Async create of unassigned node 8bd7983345d0b46ba613e0feda316ed9 with OFFLINE state 2015-04-04 10:55:40,684 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:40,684 DEBUG [main-EventThread] master.OfflineCallback(69): rs={8bd7983345d0b46ba613e0feda316ed9 state=OFFLINE, ts=1428170140674, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:40,685 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={8bd7983345d0b46ba613e0feda316ed9 state=OFFLINE, ts=1428170140674, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:40,687 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1658): localhost,52161,1428170104473 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:55:40,688 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.RegionStates(1112): Transition {8bd7983345d0b46ba613e0feda316ed9 state=OFFLINE, ts=1428170140674, server=null} to {8bd7983345d0b46ba613e0feda316ed9 state=PENDING_OPEN, ts=1428170140687, server=localhost,52161,1428170104473} 2015-04-04 10:55:40,688 INFO [PriorityRpcServer.handler=7,queue=1,port=52161] regionserver.RSRpcServices(1390): Open TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. 2015-04-04 10:55:40,694 DEBUG [RS_OPEN_REGION-localhost:52161-2] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 8bd7983345d0b46ba613e0feda316ed9 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:40,695 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1788): Bulk assigning done for localhost,52161,1428170104473 2015-04-04 10:55:40,695 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestCompactionOverride state from ENABLING to ENABLED 2015-04-04 10:55:40,696 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/8bd7983345d0b46ba613e0feda316ed9 2015-04-04 10:55:40,696 DEBUG [RS_OPEN_REGION-localhost:52161-2] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 8bd7983345d0b46ba613e0feda316ed9 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:40,697 DEBUG [RS_OPEN_REGION-localhost:52161-2] regionserver.HRegion(6078): Opening region: {ENCODED => 8bd7983345d0b46ba613e0feda316ed9, NAME => 'TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:40,697 DEBUG [AM.ZK.Worker-pool3-t4] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,52161,1428170104473, region=8bd7983345d0b46ba613e0feda316ed9, current_state={8bd7983345d0b46ba613e0feda316ed9 state=PENDING_OPEN, ts=1428170140687, server=localhost,52161,1428170104473} 2015-04-04 10:55:40,697 DEBUG [RS_OPEN_REGION-localhost:52161-2] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:40,698 DEBUG [RS_OPEN_REGION-localhost:52161-2] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:40,698 INFO [RS_OPEN_REGION-localhost:52161-2] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:55:40,697 INFO [AM.ZK.Worker-pool3-t4] master.RegionStates(1112): Transition {8bd7983345d0b46ba613e0feda316ed9 state=PENDING_OPEN, ts=1428170140687, server=localhost,52161,1428170104473} to {8bd7983345d0b46ba613e0feda316ed9 state=OPENING, ts=1428170140697, server=localhost,52161,1428170104473} 2015-04-04 10:55:40,698 INFO [RS_OPEN_REGION-localhost:52161-2] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:55:40,700 DEBUG [RS_OPEN_REGION-localhost:52161-2] coprocessor.CoprocessorHost(180): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor with path null and priority 1073741823 2015-04-04 10:55:40,700 INFO [RS_OPEN_REGION-localhost:52161-2] regionserver.RegionCoprocessorHost(350): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor from HTD of TestCompactionOverride successfully. 2015-04-04 10:55:40,700 DEBUG [RS_OPEN_REGION-localhost:52161-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestCompactionOverride 8bd7983345d0b46ba613e0feda316ed9 2015-04-04 10:55:40,700 DEBUG [RS_OPEN_REGION-localhost:52161-2] regionserver.HRegion(721): Instantiated TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. 2015-04-04 10:55:40,702 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestCompactionOverride/write-master:521580000000000 2015-04-04 10:55:40,702 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(218): Table, TestCompactionOverride, creation successful 2015-04-04 10:55:40,706 INFO [StoreOpener-8bd7983345d0b46ba613e0feda316ed9-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:40,706 INFO [StoreOpener-8bd7983345d0b46ba613e0feda316ed9-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:40,707 DEBUG [StoreOpener-8bd7983345d0b46ba613e0feda316ed9-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestCompactionOverride/8bd7983345d0b46ba613e0feda316ed9/a 2015-04-04 10:55:40,708 DEBUG [RS_OPEN_REGION-localhost:52161-2] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestCompactionOverride/8bd7983345d0b46ba613e0feda316ed9 2015-04-04 10:55:40,710 DEBUG [RS_OPEN_REGION-localhost:52161-2] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:55:40,715 DEBUG [RS_OPEN_REGION-localhost:52161-2] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestCompactionOverride/8bd7983345d0b46ba613e0feda316ed9/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:55:40,715 INFO [RS_OPEN_REGION-localhost:52161-2] regionserver.HRegion(847): Onlined 8bd7983345d0b46ba613e0feda316ed9; next sequenceid=2 2015-04-04 10:55:40,716 DEBUG [RS_OPEN_REGION-localhost:52161-2] zookeeper.ZKAssign(643): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Attempting to retransition opening state of node 8bd7983345d0b46ba613e0feda316ed9 2015-04-04 10:55:40,717 INFO [PostOpenDeployTasks:8bd7983345d0b46ba613e0feda316ed9] regionserver.HRegionServer(1849): Post open deploy tasks for TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. 2015-04-04 10:55:40,720 INFO [PostOpenDeployTasks:8bd7983345d0b46ba613e0feda316ed9] hbase.MetaTableAccessor(1350): Updated row TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. with server=localhost,52161,1428170104473 2015-04-04 10:55:40,720 DEBUG [PostOpenDeployTasks:8bd7983345d0b46ba613e0feda316ed9] regionserver.HRegionServer(1883): Finished post open deploy task for TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. 2015-04-04 10:55:40,720 DEBUG [RS_OPEN_REGION-localhost:52161-2] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 8bd7983345d0b46ba613e0feda316ed9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:40,722 DEBUG [RS_OPEN_REGION-localhost:52161-2] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 8bd7983345d0b46ba613e0feda316ed9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:40,722 DEBUG [RS_OPEN_REGION-localhost:52161-2] coordination.ZkOpenRegionCoordination(91): Transitioned 8bd7983345d0b46ba613e0feda316ed9 to OPENED in zk on localhost,52161,1428170104473 2015-04-04 10:55:40,722 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/8bd7983345d0b46ba613e0feda316ed9 2015-04-04 10:55:40,722 DEBUG [RS_OPEN_REGION-localhost:52161-2] handler.OpenRegionHandler(167): Opened TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. on localhost,52161,1428170104473 2015-04-04 10:55:40,723 DEBUG [AM.ZK.Worker-pool3-t11] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,52161,1428170104473, region=8bd7983345d0b46ba613e0feda316ed9, current_state={8bd7983345d0b46ba613e0feda316ed9 state=OPENING, ts=1428170140697, server=localhost,52161,1428170104473} 2015-04-04 10:55:40,723 INFO [AM.ZK.Worker-pool3-t11] master.RegionStates(1112): Transition {8bd7983345d0b46ba613e0feda316ed9 state=OPENING, ts=1428170140697, server=localhost,52161,1428170104473} to {8bd7983345d0b46ba613e0feda316ed9 state=OPEN, ts=1428170140723, server=localhost,52161,1428170104473} 2015-04-04 10:55:40,723 DEBUG [AM.ZK.Worker-pool3-t11] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 8bd7983345d0b46ba613e0feda316ed9 from localhost,52158,1428170104089; deleting unassigned node 2015-04-04 10:55:40,724 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/8bd7983345d0b46ba613e0feda316ed9 2015-04-04 10:55:40,724 DEBUG [AM.ZK.Worker-pool3-t11] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 8bd7983345d0b46ba613e0feda316ed9 in expected state RS_ZK_REGION_OPENED 2015-04-04 10:55:40,725 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:40,725 DEBUG [AM.ZK.Worker-pool3-t11] master.AssignmentManager$4(1337): Znode TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. deleted, state: {8bd7983345d0b46ba613e0feda316ed9 state=OPEN, ts=1428170140723, server=localhost,52161,1428170104473} 2015-04-04 10:55:40,725 DEBUG [AM.ZK.Worker-pool3-t11] master.RegionStates(460): Onlined 8bd7983345d0b46ba613e0feda316ed9 on localhost,52161,1428170104473 {ENCODED => 8bd7983345d0b46ba613e0feda316ed9, NAME => 'TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:40,880 INFO [B.defaultRpcServer.handler=1,queue=0,port=52161] regionserver.HRegion(7587): writing data to region TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. with WAL disabled. Data may be lost in the event of a crash. 2015-04-04 10:55:40,927 INFO [B.defaultRpcServer.handler=4,queue=0,port=52158] master.MasterRpcServices(648): Client=apurtell/null procedure request for: flush-table-proc 2015-04-04 10:55:40,941 INFO [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] procedure.Procedure(196): Starting procedure 'TestCompactionOverride' 2015-04-04 10:55:40,941 DEBUG [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2015-04-04 10:55:40,942 DEBUG [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] procedure.Procedure(204): Procedure 'TestCompactionOverride' starting 'acquire' 2015-04-04 10:55:40,942 DEBUG [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] procedure.Procedure(246): Starting procedure 'TestCompactionOverride', kicking off acquire phase on members. 2015-04-04 10:55:40,943 DEBUG [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(492): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestCompactionOverride 2015-04-04 10:55:40,943 DEBUG [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(93): Creating acquire znode:/hbase/flush-table-proc/acquired/TestCompactionOverride 2015-04-04 10:55:40,944 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2015-04-04 10:55:40,944 DEBUG [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(101): Watching for acquire node:/hbase/flush-table-proc/acquired/TestCompactionOverride/localhost,52161,1428170104473 2015-04-04 10:55:40,944 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2015-04-04 10:55:40,944 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-04-04 10:55:40,944 DEBUG [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(492): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestCompactionOverride/localhost,52161,1428170104473 2015-04-04 10:55:40,945 DEBUG [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2015-04-04 10:55:40,945 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/TestCompactionOverride 2015-04-04 10:55:40,945 DEBUG [main-EventThread] zookeeper.ZKUtil(492): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestCompactionOverride 2015-04-04 10:55:40,946 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2015-04-04 10:55:40,946 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/TestCompactionOverride 2015-04-04 10:55:40,946 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(151): Launching subprocedure to flush regions for TestCompactionOverride 2015-04-04 10:55:40,949 DEBUG [main-EventThread] procedure.ProcedureMember(154): Submitting new Subprocedure:TestCompactionOverride 2015-04-04 10:55:40,950 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] procedure.Subprocedure(152): Starting subprocedure 'TestCompactionOverride' with timeout 60000ms 2015-04-04 10:55:40,950 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2015-04-04 10:55:40,950 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] procedure.Subprocedure(160): Subprocedure 'TestCompactionOverride' starting 'acquire' stage 2015-04-04 10:55:40,952 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2015-04-04 10:55:40,952 DEBUG [rs(localhost,52161,1428170104473)-flush-proc-pool11-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(65): Starting region operation on TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. 2015-04-04 10:55:40,952 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(239): Waiting for local region flush to finish. 2015-04-04 10:55:40,952 DEBUG [rs(localhost,52161,1428170104473)-flush-proc-pool11-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(68): Flush region TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. started... 2015-04-04 10:55:40,953 INFO [rs(localhost,52161,1428170104473)-flush-proc-pool11-thread-1] regionserver.HRegion(2084): Started memstore flush for TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9., current region memstore size 1.64 KB, and 1/1 column families' memstores are being flushed. 2015-04-04 10:55:40,953 WARN [rs(localhost,52161,1428170104473)-flush-proc-pool11-thread-1] wal.FSHLog(1669): Couldn't find oldest seqNum for the region we are about to flush: [8bd7983345d0b46ba613e0feda316ed9] 2015-04-04 10:55:40,965 INFO [IPC Server handler 7 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741865_1041{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 5363 2015-04-04 10:55:41,369 INFO [rs(localhost,52161,1428170104473)-flush-proc-pool11-thread-1] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=24, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestCompactionOverride/8bd7983345d0b46ba613e0feda316ed9/.tmp/27c1b9309adf4c5f913b7f7a806d93f3 2015-04-04 10:55:41,381 DEBUG [rs(localhost,52161,1428170104473)-flush-proc-pool11-thread-1] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestCompactionOverride/8bd7983345d0b46ba613e0feda316ed9/.tmp/27c1b9309adf4c5f913b7f7a806d93f3 as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestCompactionOverride/8bd7983345d0b46ba613e0feda316ed9/a/27c1b9309adf4c5f913b7f7a806d93f3 2015-04-04 10:55:41,389 INFO [rs(localhost,52161,1428170104473)-flush-proc-pool11-thread-1] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestCompactionOverride/8bd7983345d0b46ba613e0feda316ed9/a/27c1b9309adf4c5f913b7f7a806d93f3, entries=10, sequenceid=24, filesize=5.2 K 2015-04-04 10:55:41,390 INFO [rs(localhost,52161,1428170104473)-flush-proc-pool11-thread-1] regionserver.HRegion(2363): Finished memstore flush of ~1.64 KB/1680, currentsize=0 B/0 for region TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. in 437ms, sequenceid=24, compaction requested=false 2015-04-04 10:55:41,390 DEBUG [rs(localhost,52161,1428170104473)-flush-proc-pool11-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. 2015-04-04 10:55:41,391 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(250): Completed 1/1 local region flush tasks. 2015-04-04 10:55:41,391 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(252): Completed 1 local region flush tasks. 2015-04-04 10:55:41,391 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(280): cancelling 0 flush region tasks localhost,52161,1428170104473 2015-04-04 10:55:41,391 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] procedure.Subprocedure(162): Subprocedure 'TestCompactionOverride' locally acquired 2015-04-04 10:55:41,391 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'localhost,52161,1428170104473' joining acquired barrier for procedure (TestCompactionOverride) in zk 2015-04-04 10:55:41,401 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestCompactionOverride/localhost,52161,1428170104473 2015-04-04 10:55:41,401 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestCompactionOverride 2015-04-04 10:55:41,401 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(186): Node created: /hbase/flush-table-proc/acquired/TestCompactionOverride/localhost,52161,1428170104473 2015-04-04 10:55:41,401 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2015-04-04 10:55:41,401 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2015-04-04 10:55:41,402 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] zookeeper.ZKUtil(492): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestCompactionOverride 2015-04-04 10:55:41,402 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] procedure.Subprocedure(167): Subprocedure 'TestCompactionOverride' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2015-04-04 10:55:41,402 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2015-04-04 10:55:41,403 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestCompactionOverride 2015-04-04 10:55:41,403 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------localhost,52161,1428170104473 2015-04-04 10:55:41,404 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2015-04-04 10:55:41,405 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2015-04-04 10:55:41,405 DEBUG [main-EventThread] procedure.Procedure(297): member: 'localhost,52161,1428170104473' joining acquired barrier for procedure 'TestCompactionOverride' on coordinator 2015-04-04 10:55:41,405 DEBUG [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] procedure.Procedure(212): Procedure 'TestCompactionOverride' starting 'in-barrier' execution. 2015-04-04 10:55:41,406 DEBUG [main-EventThread] procedure.Procedure(306): Waiting on: java.util.concurrent.CountDownLatch@17f1f18[Count = 0] remaining members to acquire global barrier 2015-04-04 10:55:41,406 DEBUG [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(115): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestCompactionOverride 2015-04-04 10:55:41,407 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestCompactionOverride 2015-04-04 10:55:41,407 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/TestCompactionOverride 2015-04-04 10:55:41,407 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Recieved reached global barrier:/hbase/flush-table-proc/reached/TestCompactionOverride 2015-04-04 10:55:41,408 DEBUG [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(492): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestCompactionOverride/localhost,52161,1428170104473 2015-04-04 10:55:41,408 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] procedure.Subprocedure(181): Subprocedure 'TestCompactionOverride' received 'reached' from coordinator. 2015-04-04 10:55:41,408 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] procedure.Subprocedure(183): Subprocedure 'TestCompactionOverride' locally completed 2015-04-04 10:55:41,408 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'TestCompactionOverride' completed for member 'localhost,52161,1428170104473' in zk 2015-04-04 10:55:41,409 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestCompactionOverride/localhost,52161,1428170104473 2015-04-04 10:55:41,409 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] procedure.Subprocedure(188): Subprocedure 'TestCompactionOverride' has notified controller of completion 2015-04-04 10:55:41,409 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(186): Node created: /hbase/flush-table-proc/reached/TestCompactionOverride/localhost,52161,1428170104473 2015-04-04 10:55:41,410 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2015-04-04 10:55:41,410 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2015-04-04 10:55:41,410 DEBUG [member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1] procedure.Subprocedure(213): Subprocedure 'TestCompactionOverride' completed. 2015-04-04 10:55:41,411 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2015-04-04 10:55:41,411 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2015-04-04 10:55:41,412 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestCompactionOverride 2015-04-04 10:55:41,412 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------localhost,52161,1428170104473 2015-04-04 10:55:41,413 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2015-04-04 10:55:41,413 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2015-04-04 10:55:41,414 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestCompactionOverride 2015-04-04 10:55:41,414 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------localhost,52161,1428170104473 2015-04-04 10:55:41,415 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(211): Finished data from procedure 'TestCompactionOverride' member 'localhost,52161,1428170104473': 2015-04-04 10:55:41,415 DEBUG [main-EventThread] procedure.Procedure(328): Member: 'localhost,52161,1428170104473' released barrier for procedure'TestCompactionOverride', counting down latch. Waiting for 0 more 2015-04-04 10:55:41,415 INFO [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] procedure.Procedure(220): Procedure 'TestCompactionOverride' execution completed 2015-04-04 10:55:41,415 DEBUG [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] procedure.Procedure(229): Running finish phase. 2015-04-04 10:55:41,416 DEBUG [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] procedure.Procedure(280): Finished coordinator procedure - removing self from list of running procedures 2015-04-04 10:55:41,416 DEBUG [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(157): Attempting to clean out zk node for op:TestCompactionOverride 2015-04-04 10:55:41,416 INFO [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureUtil(284): Clearing all znodes for procedure TestCompactionOverrideincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-04-04 10:55:41,418 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestCompactionOverride/localhost,52161,1428170104473 2015-04-04 10:55:41,418 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestCompactionOverride 2015-04-04 10:55:41,418 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2015-04-04 10:55:41,419 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2015-04-04 10:55:41,419 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-04-04 10:55:41,420 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestCompactionOverride/localhost,52161,1428170104473 2015-04-04 10:55:41,421 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestCompactionOverride 2015-04-04 10:55:41,421 DEBUG [(localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2015-04-04 10:55:41,421 INFO [B.defaultRpcServer.handler=4,queue=0,port=52158] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'TestCompactionOverride' 2015-04-04 10:55:41,422 INFO [B.defaultRpcServer.handler=4,queue=0,port=52158] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2015-04-04 10:55:41,425 DEBUG [Thread-438] client.HBaseAdmin(3381): Waiting a max of 60000 ms for procedure 'flush-table-proc : TestCompactionOverride'' to complete. (max 1714 ms per retry) 2015-04-04 10:55:41,425 DEBUG [Thread-438] client.HBaseAdmin(3390): (#1) Sleeping: 100ms while waiting for procedure completion. 2015-04-04 10:55:41,526 DEBUG [Thread-438] client.HBaseAdmin(3396): Getting current status of procedure from master... 2015-04-04 10:55:41,537 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=52158] master.MasterRpcServices(883): Checking to see if procedure from request:flush-table-proc is done 2015-04-04 10:55:41,541 DEBUG [Thread-438] coprocessor.TestRegionObserverInterface(532): Flush complete 2015-04-04 10:55:41,547 INFO [Thread-438] zookeeper.RecoverableZooKeeper(121): Process identifier=hbase-admin-on-hconnection-0x5e481993 connecting to ZooKeeper ensemble=localhost:49474 2015-04-04 10:55:41,550 DEBUG [Thread-438-EventThread] zookeeper.ZooKeeperWatcher(401): hbase-admin-on-hconnection-0x5e4819930x0, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-04-04 10:55:41,552 DEBUG [Thread-438-EventThread] zookeeper.ZooKeeperWatcher(485): hbase-admin-on-hconnection-0x5e481993-0x14c859259d50019 connected 2015-04-04 10:55:41,557 DEBUG [Thread-438] ipc.RpcClientImpl$Connection(343): Use SIMPLE authentication for service AdminService, sasl=false 2015-04-04 10:55:41,557 DEBUG [Thread-438] ipc.RpcClientImpl$Connection(708): Connecting to localhost/127.0.0.1:52161 2015-04-04 10:55:41,557 DEBUG [RpcServer.listener,port=52161] ipc.RpcServer$Listener(829): RpcServer.listener,port=52161: connection from 127.0.0.1:52275; # active connections: 6 2015-04-04 10:55:41,558 INFO [RpcServer.reader=8,bindAddress=localhost,port=52161] ipc.RpcServer$Connection(1661): Connection from 127.0.0.1 port: 52275 with version info: version: "1.1.0-SNAPSHOT" url: "git://apurtell-ltm2.internal.salesforce.com/Users/apurtell/src/hbase" revision: "5c19f9eac7d1f22147345f09aad2f04fec1ad67d" user: "apurtell" date: "Sat Apr 4 10:54:41 PDT 2015" src_checksum: "2a921de567e8e6d9e1c358f3276c1ec9" 2015-04-04 10:55:41,559 INFO [PriorityRpcServer.handler=0,queue=0,port=52161] regionserver.RSRpcServices(1046): Compacting TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. 2015-04-04 10:55:41,559 DEBUG [PriorityRpcServer.handler=0,queue=0,port=52161] compactions.RatioBasedCompactionPolicy(92): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 10 blocking 2015-04-04 10:55:41,562 DEBUG [PriorityRpcServer.handler=0,queue=0,port=52161] regionserver.HStore(1578): 8bd7983345d0b46ba613e0feda316ed9 - a: Initiating major compaction (all files) 2015-04-04 10:55:41,565 DEBUG [PriorityRpcServer.handler=0,queue=0,port=52161] regionserver.CompactSplitThread(355): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@4ec15ce; Because: User-triggered major compaction; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-04-04 10:55:41,565 INFO [RS:0;localhost:52161-shortCompactions-1428170141564] regionserver.HRegion(1770): Starting compaction on a in region TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. 2015-04-04 10:55:41,566 INFO [RS:0;localhost:52161-shortCompactions-1428170141564] regionserver.HStore(1206): Starting compaction of 1 file(s) in a of TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. into tmpdir=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestCompactionOverride/8bd7983345d0b46ba613e0feda316ed9/.tmp, totalSize=5.2 K 2015-04-04 10:55:41,569 DEBUG [RS:0;localhost:52161-shortCompactions-1428170141564] compactions.Compactor(169): Compacting hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestCompactionOverride/8bd7983345d0b46ba613e0feda316ed9/a/27c1b9309adf4c5f913b7f7a806d93f3, keycount=10, bloomtype=ROW, size=5.2 K, encoding=NONE, seqNum=24, earliestPutTs=1428170140880 2015-04-04 10:55:41,583 INFO [RS:0;localhost:52161-shortCompactions-1428170141564] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:41,593 INFO [IPC Server handler 7 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741866_1042{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 5084 2015-04-04 10:55:42,010 DEBUG [RS:0;localhost:52161-shortCompactions-1428170141564] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestCompactionOverride/8bd7983345d0b46ba613e0feda316ed9/.tmp/c52b9954b015446788689a68e775b4eb as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestCompactionOverride/8bd7983345d0b46ba613e0feda316ed9/a/c52b9954b015446788689a68e775b4eb 2015-04-04 10:55:42,034 DEBUG [RS:0;localhost:52161-shortCompactions-1428170141564] regionserver.HStore(1713): Removing store files after compaction... 2015-04-04 10:55:42,036 DEBUG [RS:0;localhost:52161-shortCompactions-1428170141564] backup.HFileArchiver(224): Archiving compacted store files. 2015-04-04 10:55:42,041 DEBUG [RS:0;localhost:52161-shortCompactions-1428170141564] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestCompactionOverride/8bd7983345d0b46ba613e0feda316ed9/a/27c1b9309adf4c5f913b7f7a806d93f3, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestCompactionOverride/8bd7983345d0b46ba613e0feda316ed9/a/27c1b9309adf4c5f913b7f7a806d93f3 2015-04-04 10:55:42,041 INFO [RS:0;localhost:52161-shortCompactions-1428170141564] regionserver.HStore(1334): Completed major compaction of 1 (all) file(s) in a of TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. into c52b9954b015446788689a68e775b4eb(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2015-04-04 10:55:42,043 INFO [RS:0;localhost:52161-shortCompactions-1428170141564] regionserver.CompactSplitThread$CompactionRunner(523): Completed compaction: Request = regionName=TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9., storeName=a, fileCount=1, fileSize=5.2 K, priority=1, time=1428170141562249000; duration=0sec 2015-04-04 10:55:42,044 DEBUG [RS:0;localhost:52161-shortCompactions-1428170141564] regionserver.CompactSplitThread$CompactionRunner(545): CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-04-04 10:55:42,572 DEBUG [Thread-438] coprocessor.TestRegionObserverInterface(543): Last compaction was at 1428170142019 2015-04-04 10:55:42,627 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testCompactionOverride Thread=470 (was 447) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t63 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: (localhost,52158,1428170104089)-proc-coordinator-pool4-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t65 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t71 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52247 [Waiting for operation #14] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t64 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t62 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS:0;localhost:52161-shortCompactions-1428170141564 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:539) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x74f1d259-metaLookup-shared--pool77-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: member: 'localhost,52161,1428170104473' subprocedure-pool3-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t69 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t108 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6313aad-shared--pool9-t12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t72 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t105 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t66 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t106 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t70 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t67 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52222 [Waiting for operation #39] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t104 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52248 [Waiting for operation #17] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t61 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t107 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t68 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=441 (was 439) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=311 (was 286) - SystemLoadAverage LEAK? -, ProcessCount=236 (was 236), AvailableMemoryMB=17 (was 18), ConnectionCount=1 (was 1) 2015-04-04 10:55:42,658 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testCheckAndPutHooks Thread=470, OpenFileDescriptor=441, MaxFileDescriptor=10240, SystemLoadAverage=311, ProcessCount=236, AvailableMemoryMB=17, ConnectionCount=1 2015-04-04 10:55:42,661 INFO [B.defaultRpcServer.handler=4,queue=0,port=52158] master.HMaster(1318): Client=apurtell/null create 'TestTable.testCheckAndPutHooks', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:55:42,665 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testCheckAndPutHooks/write-master:521580000000000 2015-04-04 10:55:42,674 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(188): Create table TestTable.testCheckAndPutHooks 2015-04-04 10:55:42,684 INFO [IPC Server handler 2 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741867_1043{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 0 2015-04-04 10:55:42,687 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testCheckAndPutHooks/.tabledesc/.tableinfo.0000000001 2015-04-04 10:55:42,689 INFO [RegionOpenAndInitThread-TestTable.testCheckAndPutHooks-1] regionserver.HRegion(5885): creating HRegion TestTable.testCheckAndPutHooks HTD == 'TestTable.testCheckAndPutHooks', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp Table name == TestTable.testCheckAndPutHooks 2015-04-04 10:55:42,697 INFO [IPC Server handler 2 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741868_1044{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 65 2015-04-04 10:55:43,099 DEBUG [RegionOpenAndInitThread-TestTable.testCheckAndPutHooks-1] regionserver.HRegion(721): Instantiated TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. 2015-04-04 10:55:43,100 DEBUG [RegionOpenAndInitThread-TestTable.testCheckAndPutHooks-1] regionserver.HRegion(1350): Closing TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674.: disabling compactions & flushes 2015-04-04 10:55:43,100 DEBUG [RegionOpenAndInitThread-TestTable.testCheckAndPutHooks-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. 2015-04-04 10:55:43,100 INFO [RegionOpenAndInitThread-TestTable.testCheckAndPutHooks-1] regionserver.HRegion(1471): Closed TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. 2015-04-04 10:55:43,105 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:55:43,107 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,52161,1428170104473 2015-04-04 10:55:43,107 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKAssign(174): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Async create of unassigned node 8875f5fffb5833b35dd5e21d2b464674 with OFFLINE state 2015-04-04 10:55:43,116 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:43,117 DEBUG [main-EventThread] master.OfflineCallback(69): rs={8875f5fffb5833b35dd5e21d2b464674 state=OFFLINE, ts=1428170143107, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:43,117 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={8875f5fffb5833b35dd5e21d2b464674 state=OFFLINE, ts=1428170143107, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:43,119 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1658): localhost,52161,1428170104473 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:55:43,119 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.RegionStates(1112): Transition {8875f5fffb5833b35dd5e21d2b464674 state=OFFLINE, ts=1428170143107, server=null} to {8875f5fffb5833b35dd5e21d2b464674 state=PENDING_OPEN, ts=1428170143119, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,120 INFO [PriorityRpcServer.handler=0,queue=0,port=52161] regionserver.RSRpcServices(1390): Open TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. 2015-04-04 10:55:43,126 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 8875f5fffb5833b35dd5e21d2b464674 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:43,127 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1788): Bulk assigning done for localhost,52161,1428170104473 2015-04-04 10:55:43,127 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testCheckAndPutHooks state from ENABLING to ENABLED 2015-04-04 10:55:43,128 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/8875f5fffb5833b35dd5e21d2b464674 2015-04-04 10:55:43,128 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 8875f5fffb5833b35dd5e21d2b464674 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:43,129 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(6078): Opening region: {ENCODED => 8875f5fffb5833b35dd5e21d2b464674, NAME => 'TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:43,129 DEBUG [AM.ZK.Worker-pool3-t16] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,52161,1428170104473, region=8875f5fffb5833b35dd5e21d2b464674, current_state={8875f5fffb5833b35dd5e21d2b464674 state=PENDING_OPEN, ts=1428170143119, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,129 INFO [AM.ZK.Worker-pool3-t16] master.RegionStates(1112): Transition {8875f5fffb5833b35dd5e21d2b464674 state=PENDING_OPEN, ts=1428170143119, server=localhost,52161,1428170104473} to {8875f5fffb5833b35dd5e21d2b464674 state=OPENING, ts=1428170143129, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,129 DEBUG [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:43,129 DEBUG [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:43,130 INFO [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:55:43,130 INFO [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:55:43,130 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testCheckAndPutHooks 8875f5fffb5833b35dd5e21d2b464674 2015-04-04 10:55:43,130 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(721): Instantiated TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. 2015-04-04 10:55:43,133 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testCheckAndPutHooks/write-master:521580000000000 2015-04-04 10:55:43,133 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(218): Table, TestTable.testCheckAndPutHooks, creation successful 2015-04-04 10:55:43,134 INFO [StoreOpener-8875f5fffb5833b35dd5e21d2b464674-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:43,135 INFO [StoreOpener-8875f5fffb5833b35dd5e21d2b464674-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:43,136 DEBUG [StoreOpener-8875f5fffb5833b35dd5e21d2b464674-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/a 2015-04-04 10:55:43,139 INFO [StoreOpener-8875f5fffb5833b35dd5e21d2b464674-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:43,139 INFO [StoreOpener-8875f5fffb5833b35dd5e21d2b464674-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:43,141 DEBUG [StoreOpener-8875f5fffb5833b35dd5e21d2b464674-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/b 2015-04-04 10:55:43,142 INFO [StoreOpener-8875f5fffb5833b35dd5e21d2b464674-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:43,143 INFO [StoreOpener-8875f5fffb5833b35dd5e21d2b464674-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:43,144 DEBUG [StoreOpener-8875f5fffb5833b35dd5e21d2b464674-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/c 2015-04-04 10:55:43,145 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674 2015-04-04 10:55:43,147 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:55:43,151 DEBUG [RS_OPEN_REGION-localhost:52161-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:55:43,151 INFO [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(847): Onlined 8875f5fffb5833b35dd5e21d2b464674; next sequenceid=2 2015-04-04 10:55:43,152 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(643): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Attempting to retransition opening state of node 8875f5fffb5833b35dd5e21d2b464674 2015-04-04 10:55:43,153 INFO [PostOpenDeployTasks:8875f5fffb5833b35dd5e21d2b464674] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. 2015-04-04 10:55:43,156 INFO [PostOpenDeployTasks:8875f5fffb5833b35dd5e21d2b464674] hbase.MetaTableAccessor(1350): Updated row TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. with server=localhost,52161,1428170104473 2015-04-04 10:55:43,157 DEBUG [PostOpenDeployTasks:8875f5fffb5833b35dd5e21d2b464674] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. 2015-04-04 10:55:43,157 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 8875f5fffb5833b35dd5e21d2b464674 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:43,159 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/8875f5fffb5833b35dd5e21d2b464674 2015-04-04 10:55:43,159 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 8875f5fffb5833b35dd5e21d2b464674 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:43,159 DEBUG [RS_OPEN_REGION-localhost:52161-0] coordination.ZkOpenRegionCoordination(91): Transitioned 8875f5fffb5833b35dd5e21d2b464674 to OPENED in zk on localhost,52161,1428170104473 2015-04-04 10:55:43,160 DEBUG [RS_OPEN_REGION-localhost:52161-0] handler.OpenRegionHandler(167): Opened TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. on localhost,52161,1428170104473 2015-04-04 10:55:43,160 DEBUG [AM.ZK.Worker-pool3-t18] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,52161,1428170104473, region=8875f5fffb5833b35dd5e21d2b464674, current_state={8875f5fffb5833b35dd5e21d2b464674 state=OPENING, ts=1428170143129, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,160 INFO [AM.ZK.Worker-pool3-t18] master.RegionStates(1112): Transition {8875f5fffb5833b35dd5e21d2b464674 state=OPENING, ts=1428170143129, server=localhost,52161,1428170104473} to {8875f5fffb5833b35dd5e21d2b464674 state=OPEN, ts=1428170143160, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,160 DEBUG [AM.ZK.Worker-pool3-t18] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 8875f5fffb5833b35dd5e21d2b464674 from localhost,52158,1428170104089; deleting unassigned node 2015-04-04 10:55:43,161 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/8875f5fffb5833b35dd5e21d2b464674 2015-04-04 10:55:43,161 DEBUG [AM.ZK.Worker-pool3-t18] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 8875f5fffb5833b35dd5e21d2b464674 in expected state RS_ZK_REGION_OPENED 2015-04-04 10:55:43,161 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:43,161 DEBUG [AM.ZK.Worker-pool3-t18] master.AssignmentManager$4(1337): Znode TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. deleted, state: {8875f5fffb5833b35dd5e21d2b464674 state=OPEN, ts=1428170143160, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,162 DEBUG [AM.ZK.Worker-pool3-t18] master.RegionStates(460): Onlined 8875f5fffb5833b35dd5e21d2b464674 on localhost,52161,1428170104473 {ENCODED => 8875f5fffb5833b35dd5e21d2b464674, NAME => 'TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:43,311 INFO [Thread-460] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:43,317 INFO [Thread-460] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:43,338 INFO [Thread-460] client.HBaseAdmin$9(979): Started disable of TestTable.testCheckAndPutHooks 2015-04-04 10:55:43,338 INFO [B.defaultRpcServer.handler=2,queue=0,port=52158] master.HMaster(1659): Client=apurtell/null disable TestTable.testCheckAndPutHooks 2015-04-04 10:55:43,341 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testCheckAndPutHooks/write-master:521580000000001 2015-04-04 10:55:43,347 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(129): Attempting to disable table TestTable.testCheckAndPutHooks 2015-04-04 10:55:43,347 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testCheckAndPutHooks state from DISABLING to DISABLING 2015-04-04 10:55:43,349 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:55:43,349 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. (offlining), current state: {8875f5fffb5833b35dd5e21d2b464674 state=OPEN, ts=1428170143162, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,350 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Creating unassigned node 8875f5fffb5833b35dd5e21d2b464674 in a CLOSING state 2015-04-04 10:55:43,350 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:43,351 INFO [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {8875f5fffb5833b35dd5e21d2b464674 state=OPEN, ts=1428170143162, server=localhost,52161,1428170104473} to {8875f5fffb5833b35dd5e21d2b464674 state=PENDING_CLOSE, ts=1428170143351, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,351 INFO [PriorityRpcServer.handler=1,queue=1,port=52161] regionserver.RSRpcServices(1018): Close 8875f5fffb5833b35dd5e21d2b464674, moving to null 2015-04-04 10:55:43,352 DEBUG [RS_CLOSE_REGION-localhost:52161-1] handler.CloseRegionHandler(119): Processing close of TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. 2015-04-04 10:55:43,352 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,52161,1428170104473 for region TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. 2015-04-04 10:55:43,353 DEBUG [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(1350): Closing TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674.: disabling compactions & flushes 2015-04-04 10:55:43,353 DEBUG [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. 2015-04-04 10:55:43,353 INFO [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(2084): Started memstore flush for TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674., current region memstore size 320 B, and 3/3 column families' memstores are being flushed. 2015-04-04 10:55:43,354 DEBUG [Thread-460] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in TestTable.testCheckAndPutHooks 2015-04-04 10:55:43,365 INFO [IPC Server handler 2 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741869_1045{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 0 2015-04-04 10:55:43,366 INFO [RS_CLOSE_REGION-localhost:52161-1] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/.tmp/4723fd47a55d4f61a27dcf568bdd89f1 2015-04-04 10:55:43,374 DEBUG [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/.tmp/4723fd47a55d4f61a27dcf568bdd89f1 as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/a/4723fd47a55d4f61a27dcf568bdd89f1 2015-04-04 10:55:43,382 INFO [RS_CLOSE_REGION-localhost:52161-1] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/a/4723fd47a55d4f61a27dcf568bdd89f1, entries=1, sequenceid=8, filesize=4.6 K 2015-04-04 10:55:43,384 INFO [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(2363): Finished memstore flush of ~320 B/320, currentsize=0 B/0 for region TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. in 31ms, sequenceid=8, compaction requested=false 2015-04-04 10:55:43,385 INFO [StoreCloserThread-TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674.-1] regionserver.HStore(879): Closed a 2015-04-04 10:55:43,386 INFO [StoreCloserThread-TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674.-1] regionserver.HStore(879): Closed b 2015-04-04 10:55:43,386 INFO [StoreCloserThread-TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674.-1] regionserver.HStore(879): Closed c 2015-04-04 10:55:43,393 DEBUG [RS_CLOSE_REGION-localhost:52161-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/recovered.edits/11.seqid to file, newSeqId=11, maxSeqId=2 2015-04-04 10:55:43,394 DEBUG [RS_CLOSE_REGION-localhost:52161-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:55:43,394 DEBUG [RS_CLOSE_REGION-localhost:52161-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:55:43,394 INFO [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(1471): Closed TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. 2015-04-04 10:55:43,394 DEBUG [RS_CLOSE_REGION-localhost:52161-1] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 8875f5fffb5833b35dd5e21d2b464674 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:43,395 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/8875f5fffb5833b35dd5e21d2b464674 2015-04-04 10:55:43,395 DEBUG [RS_CLOSE_REGION-localhost:52161-1] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 8875f5fffb5833b35dd5e21d2b464674 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:43,395 DEBUG [RS_CLOSE_REGION-localhost:52161-1] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. on localhost,52161,1428170104473 2015-04-04 10:55:43,395 DEBUG [RS_CLOSE_REGION-localhost:52161-1] handler.CloseRegionHandler(164): Closed TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. 2015-04-04 10:55:43,396 DEBUG [AM.ZK.Worker-pool3-t15] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,52161,1428170104473, region=8875f5fffb5833b35dd5e21d2b464674, current_state={8875f5fffb5833b35dd5e21d2b464674 state=PENDING_CLOSE, ts=1428170143351, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,396 DEBUG [AM.ZK.Worker-pool3-t15] handler.ClosedRegionHandler(93): Handling CLOSED event for 8875f5fffb5833b35dd5e21d2b464674 2015-04-04 10:55:43,396 DEBUG [AM.ZK.Worker-pool3-t15] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. 2015-04-04 10:55:43,398 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/8875f5fffb5833b35dd5e21d2b464674 2015-04-04 10:55:43,398 DEBUG [AM.ZK.Worker-pool3-t15] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 8875f5fffb5833b35dd5e21d2b464674 in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:55:43,398 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:43,398 INFO [AM.ZK.Worker-pool3-t15] master.RegionStates(1112): Transition {8875f5fffb5833b35dd5e21d2b464674 state=PENDING_CLOSE, ts=1428170143351, server=localhost,52161,1428170104473} to {8875f5fffb5833b35dd5e21d2b464674 state=OFFLINE, ts=1428170143398, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,398 INFO [AM.ZK.Worker-pool3-t15] master.RegionStates(600): Offlined 8875f5fffb5833b35dd5e21d2b464674 from localhost,52161,1428170104473 2015-04-04 10:55:43,404 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testCheckAndPutHooks state from DISABLING to DISABLED 2015-04-04 10:55:43,406 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(191): Disabled table, TestTable.testCheckAndPutHooks, is done=true 2015-04-04 10:55:43,407 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testCheckAndPutHooks/write-master:521580000000001 2015-04-04 10:55:43,465 INFO [Thread-460] client.HBaseAdmin(1034): Disabled TestTable.testCheckAndPutHooks 2015-04-04 10:55:43,466 INFO [B.defaultRpcServer.handler=4,queue=0,port=52158] master.HMaster(1564): Client=apurtell/null delete TestTable.testCheckAndPutHooks 2015-04-04 10:55:43,469 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testCheckAndPutHooks/write-master:521580000000002 2015-04-04 10:55:43,475 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table TestTable.testCheckAndPutHooks 2015-04-04 10:55:43,483 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:55:43,486 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => 8875f5fffb5833b35dd5e21d2b464674, NAME => 'TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:55:43,488 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(186): Archiving region TestTable.testCheckAndPutHooks,,1428170142660.8875f5fffb5833b35dd5e21d2b464674. from FS 2015-04-04 10:55:43,488 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674 2015-04-04 10:55:43,494 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/recovered.edits] 2015-04-04 10:55:43,503 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/a/4723fd47a55d4f61a27dcf568bdd89f1, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/a/4723fd47a55d4f61a27dcf568bdd89f1 2015-04-04 10:55:43,517 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/recovered.edits/11.seqid, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674/recovered.edits/11.seqid 2015-04-04 10:55:43,518 INFO [IPC Server handler 5 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741868_1044 127.0.0.1:52151 2015-04-04 10:55:43,518 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testCheckAndPutHooks/8875f5fffb5833b35dd5e21d2b464674 2015-04-04 10:55:43,519 INFO [IPC Server handler 9 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741867_1043 127.0.0.1:52151 2015-04-04 10:55:43,520 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(196): Table 'TestTable.testCheckAndPutHooks' archived! 2015-04-04 10:55:43,520 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(116): Removing 'TestTable.testCheckAndPutHooks' descriptor. 2015-04-04 10:55:43,521 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(122): Removing 'TestTable.testCheckAndPutHooks' from region states. 2015-04-04 10:55:43,521 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(126): Marking 'TestTable.testCheckAndPutHooks' as deleted. 2015-04-04 10:55:43,533 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testCheckAndPutHooks/write-master:521580000000002 2015-04-04 10:55:43,589 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=52158] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = TestTable.testCheckAndPutHooks org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testCheckAndPutHooks at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:55:43,590 INFO [Thread-460] client.HBaseAdmin(739): Deleted TestTable.testCheckAndPutHooks 2015-04-04 10:55:43,634 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testCheckAndPutHooks Thread=490 (was 470) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-89293784_1 at /127.0.0.1:52247 [Waiting for operation #17] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t78 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t118 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t110 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t112 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t79 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6313aad-shared--pool9-t13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t116 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t120 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t77 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t113 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t109 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t75 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t76 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t114 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t73 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t111 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t117 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t119 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52248 [Waiting for operation #20] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t74 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t115 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=441 (was 441), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=311 (was 311), ProcessCount=236 (was 236), AvailableMemoryMB=18 (was 17) - AvailableMemoryMB LEAK? -, ConnectionCount=1 (was 1) 2015-04-04 10:55:43,671 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testRowMutation Thread=490, OpenFileDescriptor=441, MaxFileDescriptor=10240, SystemLoadAverage=311, ProcessCount=236, AvailableMemoryMB=19, ConnectionCount=1 2015-04-04 10:55:43,674 INFO [B.defaultRpcServer.handler=3,queue=0,port=52158] master.HMaster(1318): Client=apurtell/null create 'TestTable.testRowMutation', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:55:43,677 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testRowMutation/write-master:521580000000000 2015-04-04 10:55:43,683 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(188): Create table TestTable.testRowMutation 2015-04-04 10:55:43,694 INFO [IPC Server handler 3 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741870_1046{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 0 2015-04-04 10:55:43,697 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRowMutation/.tabledesc/.tableinfo.0000000001 2015-04-04 10:55:43,699 INFO [RegionOpenAndInitThread-TestTable.testRowMutation-1] regionserver.HRegion(5885): creating HRegion TestTable.testRowMutation HTD == 'TestTable.testRowMutation', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp Table name == TestTable.testRowMutation 2015-04-04 10:55:43,709 INFO [IPC Server handler 3 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741871_1047{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 0 2015-04-04 10:55:43,710 DEBUG [RegionOpenAndInitThread-TestTable.testRowMutation-1] regionserver.HRegion(721): Instantiated TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. 2015-04-04 10:55:43,711 DEBUG [RegionOpenAndInitThread-TestTable.testRowMutation-1] regionserver.HRegion(1350): Closing TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2.: disabling compactions & flushes 2015-04-04 10:55:43,711 DEBUG [RegionOpenAndInitThread-TestTable.testRowMutation-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. 2015-04-04 10:55:43,711 INFO [RegionOpenAndInitThread-TestTable.testRowMutation-1] regionserver.HRegion(1471): Closed TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. 2015-04-04 10:55:43,716 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:55:43,717 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,52161,1428170104473 2015-04-04 10:55:43,717 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKAssign(174): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Async create of unassigned node 60ca1330af1aedaeb1e85389c3efe5c2 with OFFLINE state 2015-04-04 10:55:43,718 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:43,719 DEBUG [main-EventThread] master.OfflineCallback(69): rs={60ca1330af1aedaeb1e85389c3efe5c2 state=OFFLINE, ts=1428170143717, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:43,719 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={60ca1330af1aedaeb1e85389c3efe5c2 state=OFFLINE, ts=1428170143717, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:43,723 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1658): localhost,52161,1428170104473 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:55:43,724 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.RegionStates(1112): Transition {60ca1330af1aedaeb1e85389c3efe5c2 state=OFFLINE, ts=1428170143717, server=null} to {60ca1330af1aedaeb1e85389c3efe5c2 state=PENDING_OPEN, ts=1428170143724, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,725 INFO [PriorityRpcServer.handler=1,queue=1,port=52161] regionserver.RSRpcServices(1390): Open TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. 2015-04-04 10:55:43,731 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 60ca1330af1aedaeb1e85389c3efe5c2 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:43,731 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1788): Bulk assigning done for localhost,52161,1428170104473 2015-04-04 10:55:43,731 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testRowMutation state from ENABLING to ENABLED 2015-04-04 10:55:43,733 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/60ca1330af1aedaeb1e85389c3efe5c2 2015-04-04 10:55:43,733 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 60ca1330af1aedaeb1e85389c3efe5c2 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:43,733 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(6078): Opening region: {ENCODED => 60ca1330af1aedaeb1e85389c3efe5c2, NAME => 'TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:43,734 DEBUG [AM.ZK.Worker-pool3-t20] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,52161,1428170104473, region=60ca1330af1aedaeb1e85389c3efe5c2, current_state={60ca1330af1aedaeb1e85389c3efe5c2 state=PENDING_OPEN, ts=1428170143724, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,734 INFO [AM.ZK.Worker-pool3-t20] master.RegionStates(1112): Transition {60ca1330af1aedaeb1e85389c3efe5c2 state=PENDING_OPEN, ts=1428170143724, server=localhost,52161,1428170104473} to {60ca1330af1aedaeb1e85389c3efe5c2 state=OPENING, ts=1428170143734, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,734 DEBUG [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:43,734 DEBUG [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:43,734 INFO [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:55:43,734 INFO [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:55:43,735 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testRowMutation 60ca1330af1aedaeb1e85389c3efe5c2 2015-04-04 10:55:43,735 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(721): Instantiated TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. 2015-04-04 10:55:43,738 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testRowMutation/write-master:521580000000000 2015-04-04 10:55:43,738 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(218): Table, TestTable.testRowMutation, creation successful 2015-04-04 10:55:43,739 INFO [StoreOpener-60ca1330af1aedaeb1e85389c3efe5c2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:43,740 INFO [StoreOpener-60ca1330af1aedaeb1e85389c3efe5c2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:43,741 DEBUG [StoreOpener-60ca1330af1aedaeb1e85389c3efe5c2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/a 2015-04-04 10:55:43,743 INFO [StoreOpener-60ca1330af1aedaeb1e85389c3efe5c2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:43,743 INFO [StoreOpener-60ca1330af1aedaeb1e85389c3efe5c2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:43,744 DEBUG [StoreOpener-60ca1330af1aedaeb1e85389c3efe5c2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/b 2015-04-04 10:55:43,746 INFO [StoreOpener-60ca1330af1aedaeb1e85389c3efe5c2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:43,747 INFO [StoreOpener-60ca1330af1aedaeb1e85389c3efe5c2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:43,748 DEBUG [StoreOpener-60ca1330af1aedaeb1e85389c3efe5c2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/c 2015-04-04 10:55:43,749 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2 2015-04-04 10:55:43,751 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:55:43,756 DEBUG [RS_OPEN_REGION-localhost:52161-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:55:43,756 INFO [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(847): Onlined 60ca1330af1aedaeb1e85389c3efe5c2; next sequenceid=2 2015-04-04 10:55:43,757 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(643): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Attempting to retransition opening state of node 60ca1330af1aedaeb1e85389c3efe5c2 2015-04-04 10:55:43,758 INFO [PostOpenDeployTasks:60ca1330af1aedaeb1e85389c3efe5c2] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. 2015-04-04 10:55:43,761 INFO [PostOpenDeployTasks:60ca1330af1aedaeb1e85389c3efe5c2] hbase.MetaTableAccessor(1350): Updated row TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. with server=localhost,52161,1428170104473 2015-04-04 10:55:43,761 DEBUG [PostOpenDeployTasks:60ca1330af1aedaeb1e85389c3efe5c2] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. 2015-04-04 10:55:43,761 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 60ca1330af1aedaeb1e85389c3efe5c2 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:43,763 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/60ca1330af1aedaeb1e85389c3efe5c2 2015-04-04 10:55:43,763 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 60ca1330af1aedaeb1e85389c3efe5c2 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:43,763 DEBUG [RS_OPEN_REGION-localhost:52161-1] coordination.ZkOpenRegionCoordination(91): Transitioned 60ca1330af1aedaeb1e85389c3efe5c2 to OPENED in zk on localhost,52161,1428170104473 2015-04-04 10:55:43,763 DEBUG [RS_OPEN_REGION-localhost:52161-1] handler.OpenRegionHandler(167): Opened TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. on localhost,52161,1428170104473 2015-04-04 10:55:43,764 DEBUG [AM.ZK.Worker-pool3-t9] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,52161,1428170104473, region=60ca1330af1aedaeb1e85389c3efe5c2, current_state={60ca1330af1aedaeb1e85389c3efe5c2 state=OPENING, ts=1428170143734, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,764 INFO [AM.ZK.Worker-pool3-t9] master.RegionStates(1112): Transition {60ca1330af1aedaeb1e85389c3efe5c2 state=OPENING, ts=1428170143734, server=localhost,52161,1428170104473} to {60ca1330af1aedaeb1e85389c3efe5c2 state=OPEN, ts=1428170143764, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,764 DEBUG [AM.ZK.Worker-pool3-t9] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 60ca1330af1aedaeb1e85389c3efe5c2 from localhost,52158,1428170104089; deleting unassigned node 2015-04-04 10:55:43,765 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/60ca1330af1aedaeb1e85389c3efe5c2 2015-04-04 10:55:43,765 DEBUG [AM.ZK.Worker-pool3-t9] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 60ca1330af1aedaeb1e85389c3efe5c2 in expected state RS_ZK_REGION_OPENED 2015-04-04 10:55:43,765 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:43,765 DEBUG [AM.ZK.Worker-pool3-t9] master.AssignmentManager$4(1337): Znode TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. deleted, state: {60ca1330af1aedaeb1e85389c3efe5c2 state=OPEN, ts=1428170143764, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,766 DEBUG [AM.ZK.Worker-pool3-t9] master.RegionStates(460): Onlined 60ca1330af1aedaeb1e85389c3efe5c2 on localhost,52161,1428170104473 {ENCODED => 60ca1330af1aedaeb1e85389c3efe5c2, NAME => 'TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:43,801 INFO [Thread-475] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:43,806 INFO [Thread-475] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:43,822 INFO [Thread-475] client.HBaseAdmin$9(979): Started disable of TestTable.testRowMutation 2015-04-04 10:55:43,823 INFO [B.defaultRpcServer.handler=0,queue=0,port=52158] master.HMaster(1659): Client=apurtell/null disable TestTable.testRowMutation 2015-04-04 10:55:43,825 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testRowMutation/write-master:521580000000001 2015-04-04 10:55:43,830 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(129): Attempting to disable table TestTable.testRowMutation 2015-04-04 10:55:43,830 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testRowMutation state from DISABLING to DISABLING 2015-04-04 10:55:43,832 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:55:43,832 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. (offlining), current state: {60ca1330af1aedaeb1e85389c3efe5c2 state=OPEN, ts=1428170143766, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,833 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Creating unassigned node 60ca1330af1aedaeb1e85389c3efe5c2 in a CLOSING state 2015-04-04 10:55:43,834 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:43,834 INFO [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {60ca1330af1aedaeb1e85389c3efe5c2 state=OPEN, ts=1428170143766, server=localhost,52161,1428170104473} to {60ca1330af1aedaeb1e85389c3efe5c2 state=PENDING_CLOSE, ts=1428170143834, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,835 INFO [PriorityRpcServer.handler=2,queue=0,port=52161] regionserver.RSRpcServices(1018): Close 60ca1330af1aedaeb1e85389c3efe5c2, moving to null 2015-04-04 10:55:43,835 DEBUG [RS_CLOSE_REGION-localhost:52161-2] handler.CloseRegionHandler(119): Processing close of TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. 2015-04-04 10:55:43,835 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,52161,1428170104473 for region TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. 2015-04-04 10:55:43,836 DEBUG [Thread-475] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in TestTable.testRowMutation 2015-04-04 10:55:43,836 DEBUG [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegion(1350): Closing TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2.: disabling compactions & flushes 2015-04-04 10:55:43,837 DEBUG [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegion(1377): Updates disabled for region TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. 2015-04-04 10:55:43,837 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegion(2084): Started memstore flush for TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2., current region memstore size 960 B, and 3/3 column families' memstores are being flushed. 2015-04-04 10:55:43,849 INFO [IPC Server handler 3 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741872_1048{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 0 2015-04-04 10:55:43,850 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=6, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/.tmp/8f193d3e78724db491362fc88d0b4c0a 2015-04-04 10:55:43,866 INFO [IPC Server handler 5 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741873_1049{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 0 2015-04-04 10:55:43,867 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=6, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/.tmp/892faa49eecc4474ace8c877f87851c5 2015-04-04 10:55:43,881 INFO [IPC Server handler 9 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741874_1050{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 0 2015-04-04 10:55:43,882 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=6, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/.tmp/b2660e37fe454f8c94d68ccdf981031b 2015-04-04 10:55:43,892 DEBUG [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/.tmp/8f193d3e78724db491362fc88d0b4c0a as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/a/8f193d3e78724db491362fc88d0b4c0a 2015-04-04 10:55:43,903 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/a/8f193d3e78724db491362fc88d0b4c0a, entries=1, sequenceid=6, filesize=4.6 K 2015-04-04 10:55:43,905 DEBUG [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/.tmp/892faa49eecc4474ace8c877f87851c5 as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/b/892faa49eecc4474ace8c877f87851c5 2015-04-04 10:55:43,914 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/b/892faa49eecc4474ace8c877f87851c5, entries=1, sequenceid=6, filesize=4.6 K 2015-04-04 10:55:43,916 DEBUG [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/.tmp/b2660e37fe454f8c94d68ccdf981031b as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/c/b2660e37fe454f8c94d68ccdf981031b 2015-04-04 10:55:43,928 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/c/b2660e37fe454f8c94d68ccdf981031b, entries=1, sequenceid=6, filesize=4.6 K 2015-04-04 10:55:43,929 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegion(2363): Finished memstore flush of ~960 B/960, currentsize=0 B/0 for region TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. in 92ms, sequenceid=6, compaction requested=false 2015-04-04 10:55:43,931 INFO [StoreCloserThread-TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2.-1] regionserver.HStore(879): Closed a 2015-04-04 10:55:43,933 INFO [StoreCloserThread-TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2.-1] regionserver.HStore(879): Closed b 2015-04-04 10:55:43,934 INFO [StoreCloserThread-TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2.-1] regionserver.HStore(879): Closed c 2015-04-04 10:55:43,940 DEBUG [Thread-475] client.HBaseAdmin(1019): Sleeping= 200ms, waiting for all regions to be disabled in TestTable.testRowMutation 2015-04-04 10:55:43,941 DEBUG [RS_CLOSE_REGION-localhost:52161-2] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/recovered.edits/9.seqid to file, newSeqId=9, maxSeqId=2 2015-04-04 10:55:43,942 DEBUG [RS_CLOSE_REGION-localhost:52161-2] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:55:43,942 DEBUG [RS_CLOSE_REGION-localhost:52161-2] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:55:43,942 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegion(1471): Closed TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. 2015-04-04 10:55:43,942 DEBUG [RS_CLOSE_REGION-localhost:52161-2] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 60ca1330af1aedaeb1e85389c3efe5c2 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:43,943 DEBUG [RS_CLOSE_REGION-localhost:52161-2] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 60ca1330af1aedaeb1e85389c3efe5c2 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:43,943 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/60ca1330af1aedaeb1e85389c3efe5c2 2015-04-04 10:55:43,944 DEBUG [RS_CLOSE_REGION-localhost:52161-2] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. on localhost,52161,1428170104473 2015-04-04 10:55:43,944 DEBUG [RS_CLOSE_REGION-localhost:52161-2] handler.CloseRegionHandler(164): Closed TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. 2015-04-04 10:55:43,944 DEBUG [AM.ZK.Worker-pool3-t6] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,52161,1428170104473, region=60ca1330af1aedaeb1e85389c3efe5c2, current_state={60ca1330af1aedaeb1e85389c3efe5c2 state=PENDING_CLOSE, ts=1428170143834, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,944 DEBUG [AM.ZK.Worker-pool3-t6] handler.ClosedRegionHandler(93): Handling CLOSED event for 60ca1330af1aedaeb1e85389c3efe5c2 2015-04-04 10:55:43,944 DEBUG [AM.ZK.Worker-pool3-t6] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. 2015-04-04 10:55:43,945 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/60ca1330af1aedaeb1e85389c3efe5c2 2015-04-04 10:55:43,945 DEBUG [AM.ZK.Worker-pool3-t6] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 60ca1330af1aedaeb1e85389c3efe5c2 in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:55:43,946 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:43,946 INFO [AM.ZK.Worker-pool3-t6] master.RegionStates(1112): Transition {60ca1330af1aedaeb1e85389c3efe5c2 state=PENDING_CLOSE, ts=1428170143834, server=localhost,52161,1428170104473} to {60ca1330af1aedaeb1e85389c3efe5c2 state=OFFLINE, ts=1428170143946, server=localhost,52161,1428170104473} 2015-04-04 10:55:43,946 INFO [AM.ZK.Worker-pool3-t6] master.RegionStates(600): Offlined 60ca1330af1aedaeb1e85389c3efe5c2 from localhost,52161,1428170104473 2015-04-04 10:55:43,987 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testRowMutation state from DISABLING to DISABLED 2015-04-04 10:55:43,989 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(191): Disabled table, TestTable.testRowMutation, is done=true 2015-04-04 10:55:43,991 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testRowMutation/write-master:521580000000001 2015-04-04 10:55:44,149 INFO [Thread-475] client.HBaseAdmin(1034): Disabled TestTable.testRowMutation 2015-04-04 10:55:44,150 INFO [B.defaultRpcServer.handler=3,queue=0,port=52158] master.HMaster(1564): Client=apurtell/null delete TestTable.testRowMutation 2015-04-04 10:55:44,161 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testRowMutation/write-master:521580000000002 2015-04-04 10:55:44,166 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table TestTable.testRowMutation 2015-04-04 10:55:44,171 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:55:44,173 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => 60ca1330af1aedaeb1e85389c3efe5c2, NAME => 'TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:55:44,176 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(186): Archiving region TestTable.testRowMutation,,1428170143674.60ca1330af1aedaeb1e85389c3efe5c2. from FS 2015-04-04 10:55:44,176 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2 2015-04-04 10:55:44,181 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/recovered.edits] 2015-04-04 10:55:44,190 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/a/8f193d3e78724db491362fc88d0b4c0a, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/a/8f193d3e78724db491362fc88d0b4c0a 2015-04-04 10:55:44,196 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/b/892faa49eecc4474ace8c877f87851c5, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/b/892faa49eecc4474ace8c877f87851c5 2015-04-04 10:55:44,202 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/c/b2660e37fe454f8c94d68ccdf981031b, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/c/b2660e37fe454f8c94d68ccdf981031b 2015-04-04 10:55:44,209 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/recovered.edits/9.seqid, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2/recovered.edits/9.seqid 2015-04-04 10:55:44,210 INFO [IPC Server handler 7 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741871_1047 127.0.0.1:52151 2015-04-04 10:55:44,210 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRowMutation/60ca1330af1aedaeb1e85389c3efe5c2 2015-04-04 10:55:44,211 INFO [IPC Server handler 0 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741870_1046 127.0.0.1:52151 2015-04-04 10:55:44,211 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(196): Table 'TestTable.testRowMutation' archived! 2015-04-04 10:55:44,211 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(116): Removing 'TestTable.testRowMutation' descriptor. 2015-04-04 10:55:44,212 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(122): Removing 'TestTable.testRowMutation' from region states. 2015-04-04 10:55:44,212 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(126): Marking 'TestTable.testRowMutation' as deleted. 2015-04-04 10:55:44,218 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testRowMutation/write-master:521580000000002 2015-04-04 10:55:44,276 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52158] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = TestTable.testRowMutation org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRowMutation at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:55:44,277 INFO [Thread-475] client.HBaseAdmin(739): Deleted TestTable.testRowMutation 2015-04-04 10:55:44,314 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testRowMutation Thread=511 (was 490) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t80 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t122 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52247 [Waiting for operation #24] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t84 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t129 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6313aad-shared--pool9-t14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t86 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t125 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t123 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t126 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t132 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t121 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t124 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t83 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t87 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t128 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t85 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t81 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t127 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t131 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t130 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52285 [Waiting for operation #3] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t82 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52248 [Waiting for operation #25] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=441 (was 441), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=311 (was 311), ProcessCount=236 (was 236), AvailableMemoryMB=16 (was 19), ConnectionCount=1 (was 1) 2015-04-04 10:55:44,314 WARN [main] hbase.ResourceChecker(134): Thread=511 is superior to 500 2015-04-04 10:55:44,350 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testRegionObserver Thread=511, OpenFileDescriptor=441, MaxFileDescriptor=10240, SystemLoadAverage=311, ProcessCount=236, AvailableMemoryMB=17, ConnectionCount=1 2015-04-04 10:55:44,350 WARN [main] hbase.ResourceChecker(134): Thread=511 is superior to 500 2015-04-04 10:55:44,352 INFO [B.defaultRpcServer.handler=2,queue=0,port=52158] master.HMaster(1318): Client=apurtell/null create 'TestTable.testRegionObserver', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:55:44,364 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testRegionObserver/write-master:521580000000000 2015-04-04 10:55:44,370 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(188): Create table TestTable.testRegionObserver 2015-04-04 10:55:44,385 INFO [IPC Server handler 6 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741875_1051{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 0 2015-04-04 10:55:44,390 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRegionObserver/.tabledesc/.tableinfo.0000000001 2015-04-04 10:55:44,394 INFO [RegionOpenAndInitThread-TestTable.testRegionObserver-1] regionserver.HRegion(5885): creating HRegion TestTable.testRegionObserver HTD == 'TestTable.testRegionObserver', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp Table name == TestTable.testRegionObserver 2015-04-04 10:55:44,403 INFO [IPC Server handler 6 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741876_1052{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 63 2015-04-04 10:55:44,809 DEBUG [RegionOpenAndInitThread-TestTable.testRegionObserver-1] regionserver.HRegion(721): Instantiated TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. 2015-04-04 10:55:44,809 DEBUG [RegionOpenAndInitThread-TestTable.testRegionObserver-1] regionserver.HRegion(1350): Closing TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df.: disabling compactions & flushes 2015-04-04 10:55:44,809 DEBUG [RegionOpenAndInitThread-TestTable.testRegionObserver-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. 2015-04-04 10:55:44,809 INFO [RegionOpenAndInitThread-TestTable.testRegionObserver-1] regionserver.HRegion(1471): Closed TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. 2015-04-04 10:55:44,813 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:55:44,814 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,52161,1428170104473 2015-04-04 10:55:44,814 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKAssign(174): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Async create of unassigned node 5a6ca6a7bbc6582279a516120f00a2df with OFFLINE state 2015-04-04 10:55:44,823 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:44,823 DEBUG [main-EventThread] master.OfflineCallback(69): rs={5a6ca6a7bbc6582279a516120f00a2df state=OFFLINE, ts=1428170144814, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:44,824 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={5a6ca6a7bbc6582279a516120f00a2df state=OFFLINE, ts=1428170144814, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:44,827 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1658): localhost,52161,1428170104473 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:55:44,828 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.RegionStates(1112): Transition {5a6ca6a7bbc6582279a516120f00a2df state=OFFLINE, ts=1428170144814, server=null} to {5a6ca6a7bbc6582279a516120f00a2df state=PENDING_OPEN, ts=1428170144828, server=localhost,52161,1428170104473} 2015-04-04 10:55:44,829 INFO [PriorityRpcServer.handler=0,queue=0,port=52161] regionserver.RSRpcServices(1390): Open TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. 2015-04-04 10:55:44,835 DEBUG [RS_OPEN_REGION-localhost:52161-2] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 5a6ca6a7bbc6582279a516120f00a2df from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:44,835 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1788): Bulk assigning done for localhost,52161,1428170104473 2015-04-04 10:55:44,835 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testRegionObserver state from ENABLING to ENABLED 2015-04-04 10:55:44,836 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/5a6ca6a7bbc6582279a516120f00a2df 2015-04-04 10:55:44,836 DEBUG [RS_OPEN_REGION-localhost:52161-2] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 5a6ca6a7bbc6582279a516120f00a2df from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:44,837 DEBUG [RS_OPEN_REGION-localhost:52161-2] regionserver.HRegion(6078): Opening region: {ENCODED => 5a6ca6a7bbc6582279a516120f00a2df, NAME => 'TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:44,837 DEBUG [AM.ZK.Worker-pool3-t10] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,52161,1428170104473, region=5a6ca6a7bbc6582279a516120f00a2df, current_state={5a6ca6a7bbc6582279a516120f00a2df state=PENDING_OPEN, ts=1428170144828, server=localhost,52161,1428170104473} 2015-04-04 10:55:44,837 INFO [AM.ZK.Worker-pool3-t10] master.RegionStates(1112): Transition {5a6ca6a7bbc6582279a516120f00a2df state=PENDING_OPEN, ts=1428170144828, server=localhost,52161,1428170104473} to {5a6ca6a7bbc6582279a516120f00a2df state=OPENING, ts=1428170144837, server=localhost,52161,1428170104473} 2015-04-04 10:55:44,837 DEBUG [RS_OPEN_REGION-localhost:52161-2] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:44,837 DEBUG [RS_OPEN_REGION-localhost:52161-2] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:44,838 INFO [RS_OPEN_REGION-localhost:52161-2] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:55:44,838 INFO [RS_OPEN_REGION-localhost:52161-2] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:55:44,838 DEBUG [RS_OPEN_REGION-localhost:52161-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testRegionObserver 5a6ca6a7bbc6582279a516120f00a2df 2015-04-04 10:55:44,838 DEBUG [RS_OPEN_REGION-localhost:52161-2] regionserver.HRegion(721): Instantiated TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. 2015-04-04 10:55:44,841 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testRegionObserver/write-master:521580000000000 2015-04-04 10:55:44,841 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(218): Table, TestTable.testRegionObserver, creation successful 2015-04-04 10:55:44,841 INFO [StoreOpener-5a6ca6a7bbc6582279a516120f00a2df-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:44,842 INFO [StoreOpener-5a6ca6a7bbc6582279a516120f00a2df-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:44,843 DEBUG [StoreOpener-5a6ca6a7bbc6582279a516120f00a2df-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/a 2015-04-04 10:55:44,844 INFO [StoreOpener-5a6ca6a7bbc6582279a516120f00a2df-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:44,845 INFO [StoreOpener-5a6ca6a7bbc6582279a516120f00a2df-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:44,846 DEBUG [StoreOpener-5a6ca6a7bbc6582279a516120f00a2df-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/b 2015-04-04 10:55:44,848 INFO [StoreOpener-5a6ca6a7bbc6582279a516120f00a2df-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:44,848 INFO [StoreOpener-5a6ca6a7bbc6582279a516120f00a2df-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:44,849 DEBUG [StoreOpener-5a6ca6a7bbc6582279a516120f00a2df-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/c 2015-04-04 10:55:44,850 DEBUG [RS_OPEN_REGION-localhost:52161-2] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df 2015-04-04 10:55:44,853 DEBUG [RS_OPEN_REGION-localhost:52161-2] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:55:44,858 DEBUG [RS_OPEN_REGION-localhost:52161-2] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:55:44,858 INFO [RS_OPEN_REGION-localhost:52161-2] regionserver.HRegion(847): Onlined 5a6ca6a7bbc6582279a516120f00a2df; next sequenceid=2 2015-04-04 10:55:44,859 DEBUG [RS_OPEN_REGION-localhost:52161-2] zookeeper.ZKAssign(643): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Attempting to retransition opening state of node 5a6ca6a7bbc6582279a516120f00a2df 2015-04-04 10:55:44,860 INFO [PostOpenDeployTasks:5a6ca6a7bbc6582279a516120f00a2df] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. 2015-04-04 10:55:44,863 INFO [PostOpenDeployTasks:5a6ca6a7bbc6582279a516120f00a2df] hbase.MetaTableAccessor(1350): Updated row TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. with server=localhost,52161,1428170104473 2015-04-04 10:55:44,863 DEBUG [PostOpenDeployTasks:5a6ca6a7bbc6582279a516120f00a2df] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. 2015-04-04 10:55:44,863 DEBUG [RS_OPEN_REGION-localhost:52161-2] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 5a6ca6a7bbc6582279a516120f00a2df from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:44,865 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/5a6ca6a7bbc6582279a516120f00a2df 2015-04-04 10:55:44,865 DEBUG [RS_OPEN_REGION-localhost:52161-2] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 5a6ca6a7bbc6582279a516120f00a2df from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:44,865 DEBUG [RS_OPEN_REGION-localhost:52161-2] coordination.ZkOpenRegionCoordination(91): Transitioned 5a6ca6a7bbc6582279a516120f00a2df to OPENED in zk on localhost,52161,1428170104473 2015-04-04 10:55:44,866 DEBUG [RS_OPEN_REGION-localhost:52161-2] handler.OpenRegionHandler(167): Opened TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. on localhost,52161,1428170104473 2015-04-04 10:55:44,866 DEBUG [AM.ZK.Worker-pool3-t12] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,52161,1428170104473, region=5a6ca6a7bbc6582279a516120f00a2df, current_state={5a6ca6a7bbc6582279a516120f00a2df state=OPENING, ts=1428170144837, server=localhost,52161,1428170104473} 2015-04-04 10:55:44,866 INFO [AM.ZK.Worker-pool3-t12] master.RegionStates(1112): Transition {5a6ca6a7bbc6582279a516120f00a2df state=OPENING, ts=1428170144837, server=localhost,52161,1428170104473} to {5a6ca6a7bbc6582279a516120f00a2df state=OPEN, ts=1428170144866, server=localhost,52161,1428170104473} 2015-04-04 10:55:44,866 DEBUG [AM.ZK.Worker-pool3-t12] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 5a6ca6a7bbc6582279a516120f00a2df from localhost,52158,1428170104089; deleting unassigned node 2015-04-04 10:55:44,867 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/5a6ca6a7bbc6582279a516120f00a2df 2015-04-04 10:55:44,867 DEBUG [AM.ZK.Worker-pool3-t12] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 5a6ca6a7bbc6582279a516120f00a2df in expected state RS_ZK_REGION_OPENED 2015-04-04 10:55:44,867 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:44,868 DEBUG [AM.ZK.Worker-pool3-t12] master.AssignmentManager$4(1337): Znode TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. deleted, state: {5a6ca6a7bbc6582279a516120f00a2df state=OPEN, ts=1428170144866, server=localhost,52161,1428170104473} 2015-04-04 10:55:44,868 DEBUG [AM.ZK.Worker-pool3-t12] master.RegionStates(460): Onlined 5a6ca6a7bbc6582279a516120f00a2df on localhost,52161,1428170104473 {ENCODED => 5a6ca6a7bbc6582279a516120f00a2df, NAME => 'TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:45,006 INFO [Thread-499] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:45,011 INFO [Thread-499] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:45,025 INFO [Thread-499] client.HBaseAdmin$9(979): Started disable of TestTable.testRegionObserver 2015-04-04 10:55:45,025 INFO [B.defaultRpcServer.handler=0,queue=0,port=52158] master.HMaster(1659): Client=apurtell/null disable TestTable.testRegionObserver 2015-04-04 10:55:45,036 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testRegionObserver/write-master:521580000000001 2015-04-04 10:55:45,044 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(129): Attempting to disable table TestTable.testRegionObserver 2015-04-04 10:55:45,044 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testRegionObserver state from DISABLING to DISABLING 2015-04-04 10:55:45,045 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:55:45,046 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. (offlining), current state: {5a6ca6a7bbc6582279a516120f00a2df state=OPEN, ts=1428170144868, server=localhost,52161,1428170104473} 2015-04-04 10:55:45,047 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Creating unassigned node 5a6ca6a7bbc6582279a516120f00a2df in a CLOSING state 2015-04-04 10:55:45,048 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:45,048 INFO [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {5a6ca6a7bbc6582279a516120f00a2df state=OPEN, ts=1428170144868, server=localhost,52161,1428170104473} to {5a6ca6a7bbc6582279a516120f00a2df state=PENDING_CLOSE, ts=1428170145048, server=localhost,52161,1428170104473} 2015-04-04 10:55:45,049 INFO [PriorityRpcServer.handler=3,queue=1,port=52161] regionserver.RSRpcServices(1018): Close 5a6ca6a7bbc6582279a516120f00a2df, moving to null 2015-04-04 10:55:45,049 DEBUG [RS_CLOSE_REGION-localhost:52161-0] handler.CloseRegionHandler(119): Processing close of TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. 2015-04-04 10:55:45,049 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,52161,1428170104473 for region TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. 2015-04-04 10:55:45,050 DEBUG [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(1350): Closing TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df.: disabling compactions & flushes 2015-04-04 10:55:45,050 DEBUG [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(1377): Updates disabled for region TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. 2015-04-04 10:55:45,050 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(2084): Started memstore flush for TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df., current region memstore size 960 B, and 3/3 column families' memstores are being flushed. 2015-04-04 10:55:45,050 DEBUG [Thread-499] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in TestTable.testRegionObserver 2015-04-04 10:55:45,060 INFO [IPC Server handler 6 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741877_1053{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 0 2015-04-04 10:55:45,061 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/.tmp/00801bc71e9e4304adc6cb261c48c754 2015-04-04 10:55:45,079 INFO [IPC Server handler 0 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741878_1054{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 4726 2015-04-04 10:55:45,159 DEBUG [Thread-499] client.HBaseAdmin(1019): Sleeping= 200ms, waiting for all regions to be disabled in TestTable.testRegionObserver 2015-04-04 10:55:45,365 DEBUG [Thread-499] client.HBaseAdmin(1019): Sleeping= 300ms, waiting for all regions to be disabled in TestTable.testRegionObserver 2015-04-04 10:55:45,481 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/.tmp/9d77ff1a8f004ceda5d576d610943691 2015-04-04 10:55:45,500 INFO [IPC Server handler 5 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741879_1055{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 4726 2015-04-04 10:55:45,674 DEBUG [Thread-499] client.HBaseAdmin(1019): Sleeping= 500ms, waiting for all regions to be disabled in TestTable.testRegionObserver 2015-04-04 10:55:45,906 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/.tmp/14cad74ddd8a4b808b1f61f56a8cd30c 2015-04-04 10:55:45,916 DEBUG [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/.tmp/00801bc71e9e4304adc6cb261c48c754 as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/a/00801bc71e9e4304adc6cb261c48c754 2015-04-04 10:55:45,924 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/a/00801bc71e9e4304adc6cb261c48c754, entries=1, sequenceid=8, filesize=4.6 K 2015-04-04 10:55:45,927 DEBUG [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/.tmp/9d77ff1a8f004ceda5d576d610943691 as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/b/9d77ff1a8f004ceda5d576d610943691 2015-04-04 10:55:45,935 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/b/9d77ff1a8f004ceda5d576d610943691, entries=1, sequenceid=8, filesize=4.6 K 2015-04-04 10:55:45,936 DEBUG [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/.tmp/14cad74ddd8a4b808b1f61f56a8cd30c as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/c/14cad74ddd8a4b808b1f61f56a8cd30c 2015-04-04 10:55:45,943 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/c/14cad74ddd8a4b808b1f61f56a8cd30c, entries=1, sequenceid=8, filesize=4.6 K 2015-04-04 10:55:45,944 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(2363): Finished memstore flush of ~960 B/960, currentsize=0 B/0 for region TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. in 894ms, sequenceid=8, compaction requested=false 2015-04-04 10:55:45,946 INFO [StoreCloserThread-TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df.-1] regionserver.HStore(879): Closed a 2015-04-04 10:55:45,947 INFO [StoreCloserThread-TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df.-1] regionserver.HStore(879): Closed b 2015-04-04 10:55:45,952 INFO [StoreCloserThread-TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df.-1] regionserver.HStore(879): Closed c 2015-04-04 10:55:45,959 DEBUG [RS_CLOSE_REGION-localhost:52161-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/recovered.edits/11.seqid to file, newSeqId=11, maxSeqId=2 2015-04-04 10:55:45,960 DEBUG [RS_CLOSE_REGION-localhost:52161-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:55:45,960 DEBUG [RS_CLOSE_REGION-localhost:52161-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:55:45,960 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(1471): Closed TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. 2015-04-04 10:55:45,960 DEBUG [RS_CLOSE_REGION-localhost:52161-0] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 5a6ca6a7bbc6582279a516120f00a2df from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:45,970 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/5a6ca6a7bbc6582279a516120f00a2df 2015-04-04 10:55:45,970 DEBUG [RS_CLOSE_REGION-localhost:52161-0] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 5a6ca6a7bbc6582279a516120f00a2df from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:45,970 DEBUG [RS_CLOSE_REGION-localhost:52161-0] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. on localhost,52161,1428170104473 2015-04-04 10:55:45,970 DEBUG [RS_CLOSE_REGION-localhost:52161-0] handler.CloseRegionHandler(164): Closed TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. 2015-04-04 10:55:45,971 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,52161,1428170104473, region=5a6ca6a7bbc6582279a516120f00a2df, current_state={5a6ca6a7bbc6582279a516120f00a2df state=PENDING_CLOSE, ts=1428170145048, server=localhost,52161,1428170104473} 2015-04-04 10:55:45,971 DEBUG [AM.ZK.Worker-pool3-t14] handler.ClosedRegionHandler(93): Handling CLOSED event for 5a6ca6a7bbc6582279a516120f00a2df 2015-04-04 10:55:45,971 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. 2015-04-04 10:55:45,972 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/5a6ca6a7bbc6582279a516120f00a2df 2015-04-04 10:55:45,972 DEBUG [AM.ZK.Worker-pool3-t14] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 5a6ca6a7bbc6582279a516120f00a2df in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:55:45,972 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:45,972 INFO [AM.ZK.Worker-pool3-t14] master.RegionStates(1112): Transition {5a6ca6a7bbc6582279a516120f00a2df state=PENDING_CLOSE, ts=1428170145048, server=localhost,52161,1428170104473} to {5a6ca6a7bbc6582279a516120f00a2df state=OFFLINE, ts=1428170145972, server=localhost,52161,1428170104473} 2015-04-04 10:55:45,973 INFO [AM.ZK.Worker-pool3-t14] master.RegionStates(600): Offlined 5a6ca6a7bbc6582279a516120f00a2df from localhost,52161,1428170104473 2015-04-04 10:55:45,988 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testRegionObserver state from DISABLING to DISABLED 2015-04-04 10:55:45,990 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(191): Disabled table, TestTable.testRegionObserver, is done=true 2015-04-04 10:55:45,991 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testRegionObserver/write-master:521580000000001 2015-04-04 10:55:46,182 INFO [Thread-499] client.HBaseAdmin(1034): Disabled TestTable.testRegionObserver 2015-04-04 10:55:46,183 INFO [B.defaultRpcServer.handler=3,queue=0,port=52158] master.HMaster(1564): Client=apurtell/null delete TestTable.testRegionObserver 2015-04-04 10:55:46,185 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testRegionObserver/write-master:521580000000002 2015-04-04 10:55:46,191 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table TestTable.testRegionObserver 2015-04-04 10:55:46,195 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:55:46,198 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => 5a6ca6a7bbc6582279a516120f00a2df, NAME => 'TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:55:46,200 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(186): Archiving region TestTable.testRegionObserver,,1428170144352.5a6ca6a7bbc6582279a516120f00a2df. from FS 2015-04-04 10:55:46,200 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df 2015-04-04 10:55:46,205 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/recovered.edits] 2015-04-04 10:55:46,212 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/a/00801bc71e9e4304adc6cb261c48c754, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/a/00801bc71e9e4304adc6cb261c48c754 2015-04-04 10:55:46,219 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/b/9d77ff1a8f004ceda5d576d610943691, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/b/9d77ff1a8f004ceda5d576d610943691 2015-04-04 10:55:46,226 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/c/14cad74ddd8a4b808b1f61f56a8cd30c, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/c/14cad74ddd8a4b808b1f61f56a8cd30c 2015-04-04 10:55:46,233 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/recovered.edits/11.seqid, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df/recovered.edits/11.seqid 2015-04-04 10:55:46,234 INFO [IPC Server handler 6 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741876_1052 127.0.0.1:52151 2015-04-04 10:55:46,235 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testRegionObserver/5a6ca6a7bbc6582279a516120f00a2df 2015-04-04 10:55:46,235 INFO [IPC Server handler 4 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741875_1051 127.0.0.1:52151 2015-04-04 10:55:46,236 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(196): Table 'TestTable.testRegionObserver' archived! 2015-04-04 10:55:46,236 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(116): Removing 'TestTable.testRegionObserver' descriptor. 2015-04-04 10:55:46,236 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(122): Removing 'TestTable.testRegionObserver' from region states. 2015-04-04 10:55:46,237 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(126): Marking 'TestTable.testRegionObserver' as deleted. 2015-04-04 10:55:46,242 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testRegionObserver/write-master:521580000000002 2015-04-04 10:55:46,305 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52158] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = TestTable.testRegionObserver org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testRegionObserver at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:55:46,305 INFO [Thread-499] client.HBaseAdmin(739): Deleted TestTable.testRegionObserver 2015-04-04 10:55:46,344 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testRegionObserver Thread=537 (was 511) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t92 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t133 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t135 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t143 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t94 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52247 [Waiting for operation #30] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t137 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t141 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t89 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6313aad-shared--pool9-t15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t142 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t96 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t138 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t100 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t95 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t90 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t93 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t91 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t134 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t139 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t97 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52285 [Waiting for operation #7] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t144 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t98 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t136 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t99 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t88 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52248 [Waiting for operation #29] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t140 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=441 (was 441), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=311 (was 311), ProcessCount=236 (was 236), AvailableMemoryMB=18 (was 17) - AvailableMemoryMB LEAK? -, ConnectionCount=1 (was 1) 2015-04-04 10:55:46,344 WARN [main] hbase.ResourceChecker(134): Thread=537 is superior to 500 2015-04-04 10:55:46,378 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testIncrementHook Thread=537, OpenFileDescriptor=441, MaxFileDescriptor=10240, SystemLoadAverage=311, ProcessCount=236, AvailableMemoryMB=19, ConnectionCount=1 2015-04-04 10:55:46,378 WARN [main] hbase.ResourceChecker(134): Thread=537 is superior to 500 2015-04-04 10:55:46,381 INFO [B.defaultRpcServer.handler=2,queue=0,port=52158] master.HMaster(1318): Client=apurtell/null create 'TestTable.testIncrementHook', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:55:46,384 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testIncrementHook/write-master:521580000000000 2015-04-04 10:55:46,391 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(188): Create table TestTable.testIncrementHook 2015-04-04 10:55:46,400 INFO [IPC Server handler 7 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741880_1056{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 0 2015-04-04 10:55:46,403 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testIncrementHook/.tabledesc/.tableinfo.0000000001 2015-04-04 10:55:46,404 INFO [RegionOpenAndInitThread-TestTable.testIncrementHook-1] regionserver.HRegion(5885): creating HRegion TestTable.testIncrementHook HTD == 'TestTable.testIncrementHook', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp Table name == TestTable.testIncrementHook 2015-04-04 10:55:46,412 INFO [IPC Server handler 7 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741881_1057{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 0 2015-04-04 10:55:46,413 DEBUG [RegionOpenAndInitThread-TestTable.testIncrementHook-1] regionserver.HRegion(721): Instantiated TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. 2015-04-04 10:55:46,413 DEBUG [RegionOpenAndInitThread-TestTable.testIncrementHook-1] regionserver.HRegion(1350): Closing TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a.: disabling compactions & flushes 2015-04-04 10:55:46,413 DEBUG [RegionOpenAndInitThread-TestTable.testIncrementHook-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. 2015-04-04 10:55:46,413 INFO [RegionOpenAndInitThread-TestTable.testIncrementHook-1] regionserver.HRegion(1471): Closed TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. 2015-04-04 10:55:46,417 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:55:46,417 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,52161,1428170104473 2015-04-04 10:55:46,417 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKAssign(174): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Async create of unassigned node 4cc4a78b8474f357a989e091ab78c01a with OFFLINE state 2015-04-04 10:55:46,418 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:46,418 DEBUG [main-EventThread] master.OfflineCallback(69): rs={4cc4a78b8474f357a989e091ab78c01a state=OFFLINE, ts=1428170146417, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:46,419 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={4cc4a78b8474f357a989e091ab78c01a state=OFFLINE, ts=1428170146417, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:46,424 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1658): localhost,52161,1428170104473 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:55:46,424 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.RegionStates(1112): Transition {4cc4a78b8474f357a989e091ab78c01a state=OFFLINE, ts=1428170146417, server=null} to {4cc4a78b8474f357a989e091ab78c01a state=PENDING_OPEN, ts=1428170146424, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,424 INFO [PriorityRpcServer.handler=1,queue=1,port=52161] regionserver.RSRpcServices(1390): Open TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. 2015-04-04 10:55:46,429 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 4cc4a78b8474f357a989e091ab78c01a from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:46,429 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1788): Bulk assigning done for localhost,52161,1428170104473 2015-04-04 10:55:46,429 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testIncrementHook state from ENABLING to ENABLED 2015-04-04 10:55:46,430 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/4cc4a78b8474f357a989e091ab78c01a 2015-04-04 10:55:46,430 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 4cc4a78b8474f357a989e091ab78c01a from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:46,431 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(6078): Opening region: {ENCODED => 4cc4a78b8474f357a989e091ab78c01a, NAME => 'TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:46,431 DEBUG [AM.ZK.Worker-pool3-t2] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,52161,1428170104473, region=4cc4a78b8474f357a989e091ab78c01a, current_state={4cc4a78b8474f357a989e091ab78c01a state=PENDING_OPEN, ts=1428170146424, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,431 INFO [AM.ZK.Worker-pool3-t2] master.RegionStates(1112): Transition {4cc4a78b8474f357a989e091ab78c01a state=PENDING_OPEN, ts=1428170146424, server=localhost,52161,1428170104473} to {4cc4a78b8474f357a989e091ab78c01a state=OPENING, ts=1428170146431, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,431 DEBUG [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:46,431 DEBUG [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:46,431 INFO [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:55:46,432 INFO [RS_OPEN_REGION-localhost:52161-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:55:46,432 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testIncrementHook 4cc4a78b8474f357a989e091ab78c01a 2015-04-04 10:55:46,432 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(721): Instantiated TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. 2015-04-04 10:55:46,439 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testIncrementHook/write-master:521580000000000 2015-04-04 10:55:46,439 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(218): Table, TestTable.testIncrementHook, creation successful 2015-04-04 10:55:46,439 INFO [StoreOpener-4cc4a78b8474f357a989e091ab78c01a-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:46,439 INFO [StoreOpener-4cc4a78b8474f357a989e091ab78c01a-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:46,441 DEBUG [StoreOpener-4cc4a78b8474f357a989e091ab78c01a-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/a 2015-04-04 10:55:46,443 INFO [StoreOpener-4cc4a78b8474f357a989e091ab78c01a-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:46,443 INFO [StoreOpener-4cc4a78b8474f357a989e091ab78c01a-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:46,444 DEBUG [StoreOpener-4cc4a78b8474f357a989e091ab78c01a-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/b 2015-04-04 10:55:46,447 INFO [StoreOpener-4cc4a78b8474f357a989e091ab78c01a-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:46,447 INFO [StoreOpener-4cc4a78b8474f357a989e091ab78c01a-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:46,448 DEBUG [StoreOpener-4cc4a78b8474f357a989e091ab78c01a-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/c 2015-04-04 10:55:46,449 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a 2015-04-04 10:55:46,452 DEBUG [RS_OPEN_REGION-localhost:52161-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:55:46,456 DEBUG [RS_OPEN_REGION-localhost:52161-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:55:46,456 INFO [RS_OPEN_REGION-localhost:52161-0] regionserver.HRegion(847): Onlined 4cc4a78b8474f357a989e091ab78c01a; next sequenceid=2 2015-04-04 10:55:46,457 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(643): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Attempting to retransition opening state of node 4cc4a78b8474f357a989e091ab78c01a 2015-04-04 10:55:46,458 INFO [PostOpenDeployTasks:4cc4a78b8474f357a989e091ab78c01a] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. 2015-04-04 10:55:46,460 INFO [PostOpenDeployTasks:4cc4a78b8474f357a989e091ab78c01a] hbase.MetaTableAccessor(1350): Updated row TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. with server=localhost,52161,1428170104473 2015-04-04 10:55:46,460 DEBUG [PostOpenDeployTasks:4cc4a78b8474f357a989e091ab78c01a] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. 2015-04-04 10:55:46,461 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 4cc4a78b8474f357a989e091ab78c01a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:46,463 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/4cc4a78b8474f357a989e091ab78c01a 2015-04-04 10:55:46,463 DEBUG [RS_OPEN_REGION-localhost:52161-0] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 4cc4a78b8474f357a989e091ab78c01a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:46,463 DEBUG [RS_OPEN_REGION-localhost:52161-0] coordination.ZkOpenRegionCoordination(91): Transitioned 4cc4a78b8474f357a989e091ab78c01a to OPENED in zk on localhost,52161,1428170104473 2015-04-04 10:55:46,463 DEBUG [RS_OPEN_REGION-localhost:52161-0] handler.OpenRegionHandler(167): Opened TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. on localhost,52161,1428170104473 2015-04-04 10:55:46,463 DEBUG [AM.ZK.Worker-pool3-t3] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,52161,1428170104473, region=4cc4a78b8474f357a989e091ab78c01a, current_state={4cc4a78b8474f357a989e091ab78c01a state=OPENING, ts=1428170146431, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,463 INFO [AM.ZK.Worker-pool3-t3] master.RegionStates(1112): Transition {4cc4a78b8474f357a989e091ab78c01a state=OPENING, ts=1428170146431, server=localhost,52161,1428170104473} to {4cc4a78b8474f357a989e091ab78c01a state=OPEN, ts=1428170146463, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,463 DEBUG [AM.ZK.Worker-pool3-t3] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 4cc4a78b8474f357a989e091ab78c01a from localhost,52158,1428170104089; deleting unassigned node 2015-04-04 10:55:46,465 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/4cc4a78b8474f357a989e091ab78c01a 2015-04-04 10:55:46,465 DEBUG [AM.ZK.Worker-pool3-t3] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 4cc4a78b8474f357a989e091ab78c01a in expected state RS_ZK_REGION_OPENED 2015-04-04 10:55:46,465 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:46,465 DEBUG [AM.ZK.Worker-pool3-t20] master.AssignmentManager$4(1337): Znode TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. deleted, state: {4cc4a78b8474f357a989e091ab78c01a state=OPEN, ts=1428170146463, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,465 DEBUG [AM.ZK.Worker-pool3-t20] master.RegionStates(460): Onlined 4cc4a78b8474f357a989e091ab78c01a on localhost,52161,1428170104473 {ENCODED => 4cc4a78b8474f357a989e091ab78c01a, NAME => 'TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:46,505 INFO [Thread-522] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:46,510 INFO [Thread-522] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:46,520 INFO [Thread-522] client.HBaseAdmin$9(979): Started disable of TestTable.testIncrementHook 2015-04-04 10:55:46,520 INFO [B.defaultRpcServer.handler=4,queue=0,port=52158] master.HMaster(1659): Client=apurtell/null disable TestTable.testIncrementHook 2015-04-04 10:55:46,522 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testIncrementHook/write-master:521580000000001 2015-04-04 10:55:46,527 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(129): Attempting to disable table TestTable.testIncrementHook 2015-04-04 10:55:46,527 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testIncrementHook state from DISABLING to DISABLING 2015-04-04 10:55:46,528 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:55:46,529 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. (offlining), current state: {4cc4a78b8474f357a989e091ab78c01a state=OPEN, ts=1428170146465, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,529 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Creating unassigned node 4cc4a78b8474f357a989e091ab78c01a in a CLOSING state 2015-04-04 10:55:46,530 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:46,531 INFO [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {4cc4a78b8474f357a989e091ab78c01a state=OPEN, ts=1428170146465, server=localhost,52161,1428170104473} to {4cc4a78b8474f357a989e091ab78c01a state=PENDING_CLOSE, ts=1428170146531, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,532 INFO [PriorityRpcServer.handler=8,queue=0,port=52161] regionserver.RSRpcServices(1018): Close 4cc4a78b8474f357a989e091ab78c01a, moving to null 2015-04-04 10:55:46,532 DEBUG [RS_CLOSE_REGION-localhost:52161-1] handler.CloseRegionHandler(119): Processing close of TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. 2015-04-04 10:55:46,533 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,52161,1428170104473 for region TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. 2015-04-04 10:55:46,534 DEBUG [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(1350): Closing TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a.: disabling compactions & flushes 2015-04-04 10:55:46,534 DEBUG [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. 2015-04-04 10:55:46,534 DEBUG [Thread-522] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in TestTable.testIncrementHook 2015-04-04 10:55:46,534 INFO [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(2084): Started memstore flush for TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a., current region memstore size 168 B, and 3/3 column families' memstores are being flushed. 2015-04-04 10:55:46,546 INFO [IPC Server handler 7 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741882_1058{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 0 2015-04-04 10:55:46,547 INFO [RS_CLOSE_REGION-localhost:52161-1] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=6, memsize=168, hasBloomFilter=false, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/.tmp/d8ffac79eaf5413cba40b3ad68c45178 2015-04-04 10:55:46,556 DEBUG [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/.tmp/d8ffac79eaf5413cba40b3ad68c45178 as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/a/d8ffac79eaf5413cba40b3ad68c45178 2015-04-04 10:55:46,564 INFO [RS_CLOSE_REGION-localhost:52161-1] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/a/d8ffac79eaf5413cba40b3ad68c45178, entries=1, sequenceid=6, filesize=4.6 K 2015-04-04 10:55:46,565 INFO [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(2363): Finished memstore flush of ~168 B/168, currentsize=0 B/0 for region TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. in 31ms, sequenceid=6, compaction requested=false 2015-04-04 10:55:46,567 INFO [StoreCloserThread-TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a.-1] regionserver.HStore(879): Closed a 2015-04-04 10:55:46,567 INFO [StoreCloserThread-TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a.-1] regionserver.HStore(879): Closed b 2015-04-04 10:55:46,567 INFO [StoreCloserThread-TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a.-1] regionserver.HStore(879): Closed c 2015-04-04 10:55:46,574 DEBUG [RS_CLOSE_REGION-localhost:52161-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/recovered.edits/9.seqid to file, newSeqId=9, maxSeqId=2 2015-04-04 10:55:46,576 DEBUG [RS_CLOSE_REGION-localhost:52161-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:55:46,576 DEBUG [RS_CLOSE_REGION-localhost:52161-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:55:46,576 INFO [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(1471): Closed TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. 2015-04-04 10:55:46,577 DEBUG [RS_CLOSE_REGION-localhost:52161-1] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning 4cc4a78b8474f357a989e091ab78c01a from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:46,578 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/4cc4a78b8474f357a989e091ab78c01a 2015-04-04 10:55:46,578 DEBUG [RS_CLOSE_REGION-localhost:52161-1] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node 4cc4a78b8474f357a989e091ab78c01a from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:46,578 DEBUG [RS_CLOSE_REGION-localhost:52161-1] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. on localhost,52161,1428170104473 2015-04-04 10:55:46,579 DEBUG [RS_CLOSE_REGION-localhost:52161-1] handler.CloseRegionHandler(164): Closed TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. 2015-04-04 10:55:46,579 DEBUG [AM.ZK.Worker-pool3-t7] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,52161,1428170104473, region=4cc4a78b8474f357a989e091ab78c01a, current_state={4cc4a78b8474f357a989e091ab78c01a state=PENDING_CLOSE, ts=1428170146531, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,579 DEBUG [AM.ZK.Worker-pool3-t7] handler.ClosedRegionHandler(93): Handling CLOSED event for 4cc4a78b8474f357a989e091ab78c01a 2015-04-04 10:55:46,579 DEBUG [AM.ZK.Worker-pool3-t7] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. 2015-04-04 10:55:46,581 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/4cc4a78b8474f357a989e091ab78c01a 2015-04-04 10:55:46,581 DEBUG [AM.ZK.Worker-pool3-t7] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node 4cc4a78b8474f357a989e091ab78c01a in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:55:46,581 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:46,581 INFO [AM.ZK.Worker-pool3-t7] master.RegionStates(1112): Transition {4cc4a78b8474f357a989e091ab78c01a state=PENDING_CLOSE, ts=1428170146531, server=localhost,52161,1428170104473} to {4cc4a78b8474f357a989e091ab78c01a state=OFFLINE, ts=1428170146581, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,581 INFO [AM.ZK.Worker-pool3-t7] master.RegionStates(600): Offlined 4cc4a78b8474f357a989e091ab78c01a from localhost,52161,1428170104473 2015-04-04 10:55:46,632 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testIncrementHook state from DISABLING to DISABLED 2015-04-04 10:55:46,634 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(191): Disabled table, TestTable.testIncrementHook, is done=true 2015-04-04 10:55:46,635 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testIncrementHook/write-master:521580000000001 2015-04-04 10:55:46,641 INFO [Thread-522] client.HBaseAdmin(1034): Disabled TestTable.testIncrementHook 2015-04-04 10:55:46,642 INFO [B.defaultRpcServer.handler=1,queue=0,port=52158] master.HMaster(1564): Client=apurtell/null delete TestTable.testIncrementHook 2015-04-04 10:55:46,645 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testIncrementHook/write-master:521580000000002 2015-04-04 10:55:46,650 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table TestTable.testIncrementHook 2015-04-04 10:55:46,654 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:55:46,657 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => 4cc4a78b8474f357a989e091ab78c01a, NAME => 'TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:55:46,659 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(186): Archiving region TestTable.testIncrementHook,,1428170146380.4cc4a78b8474f357a989e091ab78c01a. from FS 2015-04-04 10:55:46,659 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a 2015-04-04 10:55:46,664 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/recovered.edits] 2015-04-04 10:55:46,670 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/a/d8ffac79eaf5413cba40b3ad68c45178, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/a/d8ffac79eaf5413cba40b3ad68c45178 2015-04-04 10:55:46,679 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/recovered.edits/9.seqid, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a/recovered.edits/9.seqid 2015-04-04 10:55:46,680 INFO [IPC Server handler 1 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741881_1057 127.0.0.1:52151 2015-04-04 10:55:46,680 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testIncrementHook/4cc4a78b8474f357a989e091ab78c01a 2015-04-04 10:55:46,681 INFO [IPC Server handler 2 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741880_1056 127.0.0.1:52151 2015-04-04 10:55:46,681 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(196): Table 'TestTable.testIncrementHook' archived! 2015-04-04 10:55:46,682 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(116): Removing 'TestTable.testIncrementHook' descriptor. 2015-04-04 10:55:46,682 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(122): Removing 'TestTable.testIncrementHook' from region states. 2015-04-04 10:55:46,682 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(126): Marking 'TestTable.testIncrementHook' as deleted. 2015-04-04 10:55:46,687 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testIncrementHook/write-master:521580000000002 2015-04-04 10:55:46,765 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=52158] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = TestTable.testIncrementHook org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testIncrementHook at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:55:46,766 INFO [Thread-522] client.HBaseAdmin(739): Deleted TestTable.testIncrementHook 2015-04-04 10:55:46,798 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testIncrementHook Thread=555 (was 537) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t104 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t150 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t145 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t155 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t153 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t101 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52247 [Waiting for operation #32] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t103 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t154 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t147 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t151 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t152 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t148 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6313aad-shared--pool9-t16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t106 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t102 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t105 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t146 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-89293784_1 at /127.0.0.1:52285 [Waiting for operation #9] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t156 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t149 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52248 [Waiting for operation #31] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=437 (was 441), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=311 (was 311), ProcessCount=236 (was 236), AvailableMemoryMB=16 (was 19), ConnectionCount=1 (was 1) 2015-04-04 10:55:46,799 WARN [main] hbase.ResourceChecker(134): Thread=555 is superior to 500 2015-04-04 10:55:46,830 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testAppendHook Thread=555, OpenFileDescriptor=437, MaxFileDescriptor=10240, SystemLoadAverage=311, ProcessCount=236, AvailableMemoryMB=19, ConnectionCount=1 2015-04-04 10:55:46,831 WARN [main] hbase.ResourceChecker(134): Thread=555 is superior to 500 2015-04-04 10:55:46,833 INFO [B.defaultRpcServer.handler=0,queue=0,port=52158] master.HMaster(1318): Client=apurtell/null create 'TestTable.testAppendHook', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:55:46,846 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testAppendHook/write-master:521580000000000 2015-04-04 10:55:46,854 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(188): Create table TestTable.testAppendHook 2015-04-04 10:55:46,863 INFO [IPC Server handler 5 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741883_1059{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 0 2015-04-04 10:55:46,866 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testAppendHook/.tabledesc/.tableinfo.0000000001 2015-04-04 10:55:46,868 INFO [RegionOpenAndInitThread-TestTable.testAppendHook-1] regionserver.HRegion(5885): creating HRegion TestTable.testAppendHook HTD == 'TestTable.testAppendHook', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp Table name == TestTable.testAppendHook 2015-04-04 10:55:46,878 INFO [IPC Server handler 5 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741884_1060{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 0 2015-04-04 10:55:46,879 DEBUG [RegionOpenAndInitThread-TestTable.testAppendHook-1] regionserver.HRegion(721): Instantiated TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. 2015-04-04 10:55:46,879 DEBUG [RegionOpenAndInitThread-TestTable.testAppendHook-1] regionserver.HRegion(1350): Closing TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e.: disabling compactions & flushes 2015-04-04 10:55:46,879 DEBUG [RegionOpenAndInitThread-TestTable.testAppendHook-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. 2015-04-04 10:55:46,879 INFO [RegionOpenAndInitThread-TestTable.testAppendHook-1] regionserver.HRegion(1471): Closed TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. 2015-04-04 10:55:46,884 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:55:46,885 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,52161,1428170104473 2015-04-04 10:55:46,885 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKAssign(174): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Async create of unassigned node e37bad39b09dd2dd58e8d425493f481e with OFFLINE state 2015-04-04 10:55:46,886 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:46,887 DEBUG [main-EventThread] master.OfflineCallback(69): rs={e37bad39b09dd2dd58e8d425493f481e state=OFFLINE, ts=1428170146885, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:46,887 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={e37bad39b09dd2dd58e8d425493f481e state=OFFLINE, ts=1428170146885, server=null}, server=localhost,52161,1428170104473 2015-04-04 10:55:46,891 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1658): localhost,52161,1428170104473 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:55:46,891 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.RegionStates(1112): Transition {e37bad39b09dd2dd58e8d425493f481e state=OFFLINE, ts=1428170146885, server=null} to {e37bad39b09dd2dd58e8d425493f481e state=PENDING_OPEN, ts=1428170146891, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,892 INFO [PriorityRpcServer.handler=1,queue=1,port=52161] regionserver.RSRpcServices(1390): Open TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. 2015-04-04 10:55:46,900 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning e37bad39b09dd2dd58e8d425493f481e from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:46,901 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] master.AssignmentManager(1788): Bulk assigning done for localhost,52161,1428170104473 2015-04-04 10:55:46,901 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testAppendHook state from ENABLING to ENABLED 2015-04-04 10:55:46,902 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node e37bad39b09dd2dd58e8d425493f481e from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:55:46,902 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/e37bad39b09dd2dd58e8d425493f481e 2015-04-04 10:55:46,902 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(6078): Opening region: {ENCODED => e37bad39b09dd2dd58e8d425493f481e, NAME => 'TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:46,903 DEBUG [AM.ZK.Worker-pool3-t4] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,52161,1428170104473, region=e37bad39b09dd2dd58e8d425493f481e, current_state={e37bad39b09dd2dd58e8d425493f481e state=PENDING_OPEN, ts=1428170146891, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,903 INFO [AM.ZK.Worker-pool3-t4] master.RegionStates(1112): Transition {e37bad39b09dd2dd58e8d425493f481e state=PENDING_OPEN, ts=1428170146891, server=localhost,52161,1428170104473} to {e37bad39b09dd2dd58e8d425493f481e state=OPENING, ts=1428170146903, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,903 DEBUG [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:46,903 DEBUG [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:55:46,903 INFO [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:55:46,904 INFO [RS_OPEN_REGION-localhost:52161-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:55:46,904 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testAppendHook e37bad39b09dd2dd58e8d425493f481e 2015-04-04 10:55:46,904 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(721): Instantiated TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. 2015-04-04 10:55:46,907 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testAppendHook/write-master:521580000000000 2015-04-04 10:55:46,907 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.CreateTableHandler(218): Table, TestTable.testAppendHook, creation successful 2015-04-04 10:55:46,908 INFO [StoreOpener-e37bad39b09dd2dd58e8d425493f481e-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:46,908 INFO [StoreOpener-e37bad39b09dd2dd58e8d425493f481e-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:46,909 DEBUG [StoreOpener-e37bad39b09dd2dd58e8d425493f481e-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/a 2015-04-04 10:55:46,911 INFO [StoreOpener-e37bad39b09dd2dd58e8d425493f481e-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:46,911 INFO [StoreOpener-e37bad39b09dd2dd58e8d425493f481e-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:46,912 DEBUG [StoreOpener-e37bad39b09dd2dd58e8d425493f481e-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/b 2015-04-04 10:55:46,914 INFO [StoreOpener-e37bad39b09dd2dd58e8d425493f481e-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:55:46,914 INFO [StoreOpener-e37bad39b09dd2dd58e8d425493f481e-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:55:46,915 DEBUG [StoreOpener-e37bad39b09dd2dd58e8d425493f481e-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/c 2015-04-04 10:55:46,916 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e 2015-04-04 10:55:46,918 DEBUG [RS_OPEN_REGION-localhost:52161-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:55:46,921 DEBUG [RS_OPEN_REGION-localhost:52161-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:55:46,922 INFO [RS_OPEN_REGION-localhost:52161-1] regionserver.HRegion(847): Onlined e37bad39b09dd2dd58e8d425493f481e; next sequenceid=2 2015-04-04 10:55:46,923 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(643): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Attempting to retransition opening state of node e37bad39b09dd2dd58e8d425493f481e 2015-04-04 10:55:46,924 INFO [PostOpenDeployTasks:e37bad39b09dd2dd58e8d425493f481e] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. 2015-04-04 10:55:46,926 INFO [PostOpenDeployTasks:e37bad39b09dd2dd58e8d425493f481e] hbase.MetaTableAccessor(1350): Updated row TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. with server=localhost,52161,1428170104473 2015-04-04 10:55:46,926 DEBUG [PostOpenDeployTasks:e37bad39b09dd2dd58e8d425493f481e] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. 2015-04-04 10:55:46,929 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning e37bad39b09dd2dd58e8d425493f481e from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:46,930 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/e37bad39b09dd2dd58e8d425493f481e 2015-04-04 10:55:46,930 DEBUG [RS_OPEN_REGION-localhost:52161-1] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node e37bad39b09dd2dd58e8d425493f481e from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:55:46,930 DEBUG [RS_OPEN_REGION-localhost:52161-1] coordination.ZkOpenRegionCoordination(91): Transitioned e37bad39b09dd2dd58e8d425493f481e to OPENED in zk on localhost,52161,1428170104473 2015-04-04 10:55:46,930 DEBUG [RS_OPEN_REGION-localhost:52161-1] handler.OpenRegionHandler(167): Opened TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. on localhost,52161,1428170104473 2015-04-04 10:55:46,931 DEBUG [AM.ZK.Worker-pool3-t11] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,52161,1428170104473, region=e37bad39b09dd2dd58e8d425493f481e, current_state={e37bad39b09dd2dd58e8d425493f481e state=OPENING, ts=1428170146903, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,931 INFO [AM.ZK.Worker-pool3-t11] master.RegionStates(1112): Transition {e37bad39b09dd2dd58e8d425493f481e state=OPENING, ts=1428170146903, server=localhost,52161,1428170104473} to {e37bad39b09dd2dd58e8d425493f481e state=OPEN, ts=1428170146931, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,931 DEBUG [AM.ZK.Worker-pool3-t11] coordination.ZkOpenRegionCoordination(350): Handling OPENED of e37bad39b09dd2dd58e8d425493f481e from localhost,52158,1428170104089; deleting unassigned node 2015-04-04 10:55:46,932 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/e37bad39b09dd2dd58e8d425493f481e 2015-04-04 10:55:46,932 DEBUG [AM.ZK.Worker-pool3-t11] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node e37bad39b09dd2dd58e8d425493f481e in expected state RS_ZK_REGION_OPENED 2015-04-04 10:55:46,932 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:46,932 DEBUG [AM.ZK.Worker-pool3-t11] master.AssignmentManager$4(1337): Znode TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. deleted, state: {e37bad39b09dd2dd58e8d425493f481e state=OPEN, ts=1428170146931, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,932 DEBUG [AM.ZK.Worker-pool3-t11] master.RegionStates(460): Onlined e37bad39b09dd2dd58e8d425493f481e on localhost,52161,1428170104473 {ENCODED => e37bad39b09dd2dd58e8d425493f481e, NAME => 'TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:55:46,973 INFO [Thread-537] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:46,978 INFO [Thread-537] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:55:46,985 INFO [Thread-537] client.HBaseAdmin$9(979): Started disable of TestTable.testAppendHook 2015-04-04 10:55:46,985 INFO [B.defaultRpcServer.handler=3,queue=0,port=52158] master.HMaster(1659): Client=apurtell/null disable TestTable.testAppendHook 2015-04-04 10:55:46,987 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testAppendHook/write-master:521580000000001 2015-04-04 10:55:46,993 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(129): Attempting to disable table TestTable.testAppendHook 2015-04-04 10:55:46,993 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testAppendHook state from DISABLING to DISABLING 2015-04-04 10:55:46,994 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:55:46,995 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. (offlining), current state: {e37bad39b09dd2dd58e8d425493f481e state=OPEN, ts=1428170146932, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,995 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Creating unassigned node e37bad39b09dd2dd58e8d425493f481e in a CLOSING state 2015-04-04 10:55:46,996 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:46,997 INFO [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {e37bad39b09dd2dd58e8d425493f481e state=OPEN, ts=1428170146932, server=localhost,52161,1428170104473} to {e37bad39b09dd2dd58e8d425493f481e state=PENDING_CLOSE, ts=1428170146997, server=localhost,52161,1428170104473} 2015-04-04 10:55:46,997 INFO [PriorityRpcServer.handler=9,queue=1,port=52161] regionserver.RSRpcServices(1018): Close e37bad39b09dd2dd58e8d425493f481e, moving to null 2015-04-04 10:55:46,998 DEBUG [RS_CLOSE_REGION-localhost:52161-2] handler.CloseRegionHandler(119): Processing close of TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. 2015-04-04 10:55:46,998 DEBUG [localhost,52158,1428170104089-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,52161,1428170104473 for region TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. 2015-04-04 10:55:46,999 DEBUG [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegion(1350): Closing TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e.: disabling compactions & flushes 2015-04-04 10:55:46,999 DEBUG [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegion(1377): Updates disabled for region TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. 2015-04-04 10:55:46,999 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegion(2084): Started memstore flush for TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e., current region memstore size 160 B, and 3/3 column families' memstores are being flushed. 2015-04-04 10:55:47,000 DEBUG [Thread-537] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in TestTable.testAppendHook 2015-04-04 10:55:47,011 INFO [IPC Server handler 5 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741885_1061{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 0 2015-04-04 10:55:47,012 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=6, memsize=160, hasBloomFilter=false, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/.tmp/b6414ba4dd4c49e1910066a52795296d 2015-04-04 10:55:47,021 DEBUG [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/.tmp/b6414ba4dd4c49e1910066a52795296d as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/a/b6414ba4dd4c49e1910066a52795296d 2015-04-04 10:55:47,030 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/a/b6414ba4dd4c49e1910066a52795296d, entries=1, sequenceid=6, filesize=4.6 K 2015-04-04 10:55:47,031 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegion(2363): Finished memstore flush of ~160 B/160, currentsize=0 B/0 for region TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. in 32ms, sequenceid=6, compaction requested=false 2015-04-04 10:55:47,033 INFO [StoreCloserThread-TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e.-1] regionserver.HStore(879): Closed a 2015-04-04 10:55:47,033 INFO [StoreCloserThread-TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e.-1] regionserver.HStore(879): Closed b 2015-04-04 10:55:47,033 INFO [StoreCloserThread-TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e.-1] regionserver.HStore(879): Closed c 2015-04-04 10:55:47,039 DEBUG [RS_CLOSE_REGION-localhost:52161-2] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/recovered.edits/9.seqid to file, newSeqId=9, maxSeqId=2 2015-04-04 10:55:47,041 DEBUG [RS_CLOSE_REGION-localhost:52161-2] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:55:47,041 DEBUG [RS_CLOSE_REGION-localhost:52161-2] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:55:47,041 INFO [RS_CLOSE_REGION-localhost:52161-2] regionserver.HRegion(1471): Closed TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. 2015-04-04 10:55:47,041 DEBUG [RS_CLOSE_REGION-localhost:52161-2] zookeeper.ZKAssign(805): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioning e37bad39b09dd2dd58e8d425493f481e from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:47,043 DEBUG [RS_CLOSE_REGION-localhost:52161-2] zookeeper.ZKAssign(880): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Transitioned node e37bad39b09dd2dd58e8d425493f481e from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:55:47,043 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/e37bad39b09dd2dd58e8d425493f481e 2015-04-04 10:55:47,043 DEBUG [RS_CLOSE_REGION-localhost:52161-2] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. on localhost,52161,1428170104473 2015-04-04 10:55:47,043 DEBUG [RS_CLOSE_REGION-localhost:52161-2] handler.CloseRegionHandler(164): Closed TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. 2015-04-04 10:55:47,043 DEBUG [AM.ZK.Worker-pool3-t17] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,52161,1428170104473, region=e37bad39b09dd2dd58e8d425493f481e, current_state={e37bad39b09dd2dd58e8d425493f481e state=PENDING_CLOSE, ts=1428170146997, server=localhost,52161,1428170104473} 2015-04-04 10:55:47,044 DEBUG [AM.ZK.Worker-pool3-t17] handler.ClosedRegionHandler(93): Handling CLOSED event for e37bad39b09dd2dd58e8d425493f481e 2015-04-04 10:55:47,044 DEBUG [AM.ZK.Worker-pool3-t17] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. 2015-04-04 10:55:47,045 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/e37bad39b09dd2dd58e8d425493f481e 2015-04-04 10:55:47,045 DEBUG [AM.ZK.Worker-pool3-t17] zookeeper.ZKAssign(479): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Deleted unassigned node e37bad39b09dd2dd58e8d425493f481e in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:55:47,045 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:55:47,045 INFO [AM.ZK.Worker-pool3-t17] master.RegionStates(1112): Transition {e37bad39b09dd2dd58e8d425493f481e state=PENDING_CLOSE, ts=1428170146997, server=localhost,52161,1428170104473} to {e37bad39b09dd2dd58e8d425493f481e state=OFFLINE, ts=1428170147045, server=localhost,52161,1428170104473} 2015-04-04 10:55:47,045 INFO [AM.ZK.Worker-pool3-t17] master.RegionStates(600): Offlined e37bad39b09dd2dd58e8d425493f481e from localhost,52161,1428170104473 2015-04-04 10:55:47,046 WARN [MASTER_TABLE_OPERATIONS-localhost:52158-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testAppendHook state from DISABLING to DISABLED 2015-04-04 10:55:47,047 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DisableTableHandler(191): Disabled table, TestTable.testAppendHook, is done=true 2015-04-04 10:55:47,048 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testAppendHook/write-master:521580000000001 2015-04-04 10:55:47,114 INFO [Thread-537] client.HBaseAdmin(1034): Disabled TestTable.testAppendHook 2015-04-04 10:55:47,115 INFO [B.defaultRpcServer.handler=0,queue=0,port=52158] master.HMaster(1564): Client=apurtell/null delete TestTable.testAppendHook 2015-04-04 10:55:47,116 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=52158] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testAppendHook/write-master:521580000000002 2015-04-04 10:55:47,121 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table TestTable.testAppendHook 2015-04-04 10:55:47,126 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:55:47,128 INFO [MASTER_TABLE_OPERATIONS-localhost:52158-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => e37bad39b09dd2dd58e8d425493f481e, NAME => 'TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:55:47,130 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(186): Archiving region TestTable.testAppendHook,,1428170146833.e37bad39b09dd2dd58e8d425493f481e. from FS 2015-04-04 10:55:47,131 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e 2015-04-04 10:55:47,135 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/recovered.edits] 2015-04-04 10:55:47,141 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/a/b6414ba4dd4c49e1910066a52795296d, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/a/b6414ba4dd4c49e1910066a52795296d 2015-04-04 10:55:47,150 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/recovered.edits/9.seqid, to hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/archive/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e/recovered.edits/9.seqid 2015-04-04 10:55:47,150 INFO [IPC Server handler 0 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741884_1060 127.0.0.1:52151 2015-04-04 10:55:47,151 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/.tmp/data/default/TestTable.testAppendHook/e37bad39b09dd2dd58e8d425493f481e 2015-04-04 10:55:47,151 INFO [IPC Server handler 3 on 52150] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741883_1059 127.0.0.1:52151 2015-04-04 10:55:47,152 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(196): Table 'TestTable.testAppendHook' archived! 2015-04-04 10:55:47,152 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(116): Removing 'TestTable.testAppendHook' descriptor. 2015-04-04 10:55:47,153 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(122): Removing 'TestTable.testAppendHook' from region states. 2015-04-04 10:55:47,153 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] handler.DeleteTableHandler(126): Marking 'TestTable.testAppendHook' as deleted. 2015-04-04 10:55:47,157 DEBUG [MASTER_TABLE_OPERATIONS-localhost:52158-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testAppendHook/write-master:521580000000002 2015-04-04 10:55:47,235 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=52158] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = TestTable.testAppendHook org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestTable.testAppendHook at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:55:47,235 INFO [Thread-537] client.HBaseAdmin(739): Deleted TestTable.testAppendHook 2015-04-04 10:55:47,268 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testAppendHook Thread=574 (was 555) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t167 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t111 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t112 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52247 [Waiting for operation #34] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t158 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t159 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t166 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t168 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6313aad-shared--pool9-t17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t164 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t109 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t108 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t163 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t157 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t165 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t162 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t110 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-89293784_1 at /127.0.0.1:52285 [Waiting for operation #11] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x5e481993-shared--pool14-t107 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t161 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3723a913-shared--pool7-t160 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-243361319_1 at /127.0.0.1:52248 [Waiting for operation #33] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=437 (was 437), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=311 (was 311), ProcessCount=236 (was 236), AvailableMemoryMB=18 (was 19), ConnectionCount=1 (was 1) 2015-04-04 10:55:47,269 WARN [main] hbase.ResourceChecker(134): Thread=574 is superior to 500 2015-04-04 10:55:47,269 INFO [main] hbase.HBaseTestingUtility(1049): Shutting down minicluster 2015-04-04 10:55:47,269 INFO [main] client.ConnectionManager$HConnectionImplementation(2055): Closing master protocol: MasterService 2015-04-04 10:55:47,274 INFO [main] client.ConnectionManager$HConnectionImplementation(1675): Closing zookeeper sessionid=0x14c859259d50008 2015-04-04 10:55:47,283 DEBUG [main] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:55:47,284 DEBUG [RpcServer.reader=2,bindAddress=localhost,port=52158] ipc.RpcServer$Listener(861): RpcServer.listener,port=52158: DISCONNECTING client 127.0.0.1:52187 because read count=-1. Number of active connections: 2 2015-04-04 10:55:47,284 DEBUG [RpcServer.reader=5,bindAddress=localhost,port=52161] ipc.RpcServer$Listener(861): RpcServer.listener,port=52161: DISCONNECTING client 127.0.0.1:52188 because read count=-1. Number of active connections: 6 2015-04-04 10:55:47,284 DEBUG [RpcServer.reader=8,bindAddress=localhost,port=52161] ipc.RpcServer$Listener(861): RpcServer.listener,port=52161: DISCONNECTING client 127.0.0.1:52275 because read count=-1. Number of active connections: 6 2015-04-04 10:55:47,388 DEBUG [main] util.JVMClusterUtil(241): Shutting down HBase Cluster 2015-04-04 10:55:47,388 INFO [main] regionserver.HRegionServer(1823): STOPPED: Cluster shutdown requested 2015-04-04 10:55:47,389 INFO [M:0;localhost:52158] regionserver.HRegionServer(955): Stopping infoServer 2015-04-04 10:55:47,393 INFO [M:0;localhost:52158] log.Slf4jLog(67): Stopped SelectChannelConnector@0.0.0.0:0 2015-04-04 10:55:47,395 INFO [M:0;localhost:52158] regionserver.HRegionServer(995): stopping server localhost,52158,1428170104089 2015-04-04 10:55:47,395 DEBUG [M:0;localhost:52158] zookeeper.MetaTableLocator(619): Stopping MetaTableLocator 2015-04-04 10:55:47,399 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2015-04-04 10:55:47,399 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2015-04-04 10:55:47,399 INFO [main] regionserver.HRegionServer(1823): STOPPED: Shutdown requested 2015-04-04 10:55:47,399 INFO [RS:0;localhost:52161] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-04-04 10:55:47,400 INFO [RS:0;localhost:52161] regionserver.HRegionServer(955): Stopping infoServer 2015-04-04 10:55:47,400 INFO [SplitLogWorker-localhost:52161] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-04-04 10:55:47,400 INFO [SplitLogWorker-localhost:52161] regionserver.SplitLogWorker(155): SplitLogWorker localhost,52161,1428170104473 exiting 2015-04-04 10:55:47,400 DEBUG [main-EventThread] zookeeper.ZKUtil(492): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2015-04-04 10:55:47,400 DEBUG [main-EventThread] zookeeper.ZKUtil(492): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2015-04-04 10:55:47,404 INFO [RS:0;localhost:52161] log.Slf4jLog(67): Stopped SelectChannelConnector@0.0.0.0:0 2015-04-04 10:55:47,406 INFO [RS:0;localhost:52161] regionserver.HeapMemoryManager(198): Stoping HeapMemoryTuner chore. 2015-04-04 10:55:47,406 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-04-04 10:55:47,406 INFO [RS:0;localhost:52161] snapshot.RegionServerSnapshotManager(134): Stopping RegionServerSnapshotManager gracefully. 2015-04-04 10:55:47,407 INFO [RS:0;localhost:52161] flush.RegionServerFlushTableProcedureManager(113): Stopping region server flush procedure manager gracefully. 2015-04-04 10:55:47,407 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-04-04 10:55:47,408 DEBUG [RS_CLOSE_REGION-localhost:52161-0] handler.CloseRegionHandler(119): Processing close of hbase:namespace,,1428170106679.91f8eee82cc828ca62036b3635da92af. 2015-04-04 10:55:47,408 DEBUG [RS_CLOSE_REGION-localhost:52161-1] handler.CloseRegionHandler(119): Processing close of TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. 2015-04-04 10:55:47,408 INFO [RS:0;localhost:52161] regionserver.HRegionServer(995): stopping server localhost,52161,1428170104473 2015-04-04 10:55:47,409 DEBUG [RS:0;localhost:52161] zookeeper.MetaTableLocator(619): Stopping MetaTableLocator 2015-04-04 10:55:47,408 DEBUG [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(1350): Closing TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9.: disabling compactions & flushes 2015-04-04 10:55:47,409 DEBUG [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(1377): Updates disabled for region TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. 2015-04-04 10:55:47,408 DEBUG [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(1350): Closing hbase:namespace,,1428170106679.91f8eee82cc828ca62036b3635da92af.: disabling compactions & flushes 2015-04-04 10:55:47,409 DEBUG [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(1377): Updates disabled for region hbase:namespace,,1428170106679.91f8eee82cc828ca62036b3635da92af. 2015-04-04 10:55:47,409 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(2084): Started memstore flush for hbase:namespace,,1428170106679.91f8eee82cc828ca62036b3635da92af., current region memstore size 344 B, and 1/1 column families' memstores are being flushed. 2015-04-04 10:55:47,410 INFO [StoreCloserThread-TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9.-1] regionserver.HStore(879): Closed a 2015-04-04 10:55:47,411 INFO [RS:0;localhost:52161] client.ConnectionManager$HConnectionImplementation(1675): Closing zookeeper sessionid=0x14c859259d50002 2015-04-04 10:55:47,411 INFO [M:0;localhost:52158] client.ConnectionManager$HConnectionImplementation(1675): Closing zookeeper sessionid=0x14c859259d50003 2015-04-04 10:55:47,412 DEBUG [RS:0;localhost:52161] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:55:47,412 DEBUG [RpcServer.reader=3,bindAddress=localhost,port=52161] ipc.RpcServer$Listener(861): RpcServer.listener,port=52161: DISCONNECTING client 127.0.0.1:52183 because read count=-1. Number of active connections: 4 2015-04-04 10:55:47,412 DEBUG [M:0;localhost:52158] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:55:47,413 DEBUG [RpcServer.reader=2,bindAddress=localhost,port=52161] ipc.RpcServer$Listener(861): RpcServer.listener,port=52161: DISCONNECTING client 127.0.0.1:52180 because read count=-1. Number of active connections: 3 2015-04-04 10:55:47,416 INFO [localhost,52161,1428170104473_ChoreService_1] hbase.ScheduledChore(179): Chore: localhost,52161,1428170104473-MemstoreFlusherChore was stopped 2015-04-04 10:55:47,418 DEBUG [RS_CLOSE_REGION-localhost:52161-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/default/TestCompactionOverride/8bd7983345d0b46ba613e0feda316ed9/recovered.edits/28.seqid to file, newSeqId=28, maxSeqId=2 2015-04-04 10:55:47,420 DEBUG [RS_CLOSE_REGION-localhost:52161-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:55:47,420 DEBUG [RS_CLOSE_REGION-localhost:52161-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:55:47,420 DEBUG [RS_CLOSE_REGION-localhost:52161-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor 2015-04-04 10:55:47,420 INFO [RS_CLOSE_REGION-localhost:52161-1] regionserver.HRegion(1471): Closed TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. 2015-04-04 10:55:47,420 DEBUG [RS_CLOSE_REGION-localhost:52161-1] handler.CloseRegionHandler(164): Closed TestCompactionOverride,,1428170140227.8bd7983345d0b46ba613e0feda316ed9. 2015-04-04 10:55:47,422 INFO [IPC Server handler 8 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741886_1062{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 0 2015-04-04 10:55:47,423 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=344, hasBloomFilter=true, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/hbase/namespace/91f8eee82cc828ca62036b3635da92af/.tmp/c2bdb501aadc44458a5f938c1c66362f 2015-04-04 10:55:47,430 INFO [RS_OPEN_META-localhost:52161-0-MetaLogRoller] regionserver.LogRoller(163): LogRoller exiting. 2015-04-04 10:55:47,431 DEBUG [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/hbase/namespace/91f8eee82cc828ca62036b3635da92af/.tmp/c2bdb501aadc44458a5f938c1c66362f as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/hbase/namespace/91f8eee82cc828ca62036b3635da92af/info/c2bdb501aadc44458a5f938c1c66362f 2015-04-04 10:55:47,440 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/hbase/namespace/91f8eee82cc828ca62036b3635da92af/info/c2bdb501aadc44458a5f938c1c66362f, entries=2, sequenceid=8, filesize=4.9 K 2015-04-04 10:55:47,441 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(2363): Finished memstore flush of ~344 B/344, currentsize=0 B/0 for region hbase:namespace,,1428170106679.91f8eee82cc828ca62036b3635da92af. in 32ms, sequenceid=8, compaction requested=false 2015-04-04 10:55:47,442 INFO [StoreCloserThread-hbase:namespace,,1428170106679.91f8eee82cc828ca62036b3635da92af.-1] regionserver.HStore(879): Closed info 2015-04-04 10:55:47,449 DEBUG [RS_CLOSE_REGION-localhost:52161-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/hbase/namespace/91f8eee82cc828ca62036b3635da92af/recovered.edits/11.seqid to file, newSeqId=11, maxSeqId=2 2015-04-04 10:55:47,450 DEBUG [RS_CLOSE_REGION-localhost:52161-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:55:47,450 DEBUG [RS_CLOSE_REGION-localhost:52161-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:55:47,450 INFO [RS_CLOSE_REGION-localhost:52161-0] regionserver.HRegion(1471): Closed hbase:namespace,,1428170106679.91f8eee82cc828ca62036b3635da92af. 2015-04-04 10:55:47,450 DEBUG [RS_CLOSE_REGION-localhost:52161-0] handler.CloseRegionHandler(164): Closed hbase:namespace,,1428170106679.91f8eee82cc828ca62036b3635da92af. 2015-04-04 10:55:47,515 INFO [regionserver/localhost/127.0.0.1:0.logRoller] regionserver.LogRoller(163): LogRoller exiting. 2015-04-04 10:55:47,515 INFO [regionserver/localhost/127.0.0.1:0.leaseChecker] regionserver.Leases(146): regionserver/localhost/127.0.0.1:0.leaseChecker closing leases 2015-04-04 10:55:47,515 INFO [regionserver/localhost/127.0.0.1:0.leaseChecker] regionserver.Leases(149): regionserver/localhost/127.0.0.1:0.leaseChecker closed leases 2015-04-04 10:55:47,516 INFO [RS:0;localhost:52161] regionserver.CompactSplitThread(393): Waiting for Split Thread to finish... 2015-04-04 10:55:47,517 INFO [M:0;localhost:52158] regionserver.HRegionServer(1023): stopping server localhost,52158,1428170104089; all regions closed. 2015-04-04 10:55:47,517 INFO [RS:0;localhost:52161] regionserver.CompactSplitThread(393): Waiting for Merge Thread to finish... 2015-04-04 10:55:47,517 INFO [RS:0;localhost:52161] regionserver.CompactSplitThread(393): Waiting for Large Compaction Thread to finish... 2015-04-04 10:55:47,517 INFO [RS:0;localhost:52161] regionserver.CompactSplitThread(393): Waiting for Small Compaction Thread to finish... 2015-04-04 10:55:47,517 INFO [M:0;localhost:52158] hbase.ChoreService(303): Chore service for: localhost,52158,1428170104089 had [java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3e4ce243, java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@299abca9, java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@6dd81095, java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@428de8a, java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@172a3828] on shutdown 2015-04-04 10:55:47,518 INFO [M:0;localhost:52158] master.ServerManager(546): Waiting on regionserver(s) to go down localhost,52161,1428170104473 2015-04-04 10:55:47,518 INFO [RS:0;localhost:52161] regionserver.HRegionServer(1217): Waiting on 1 regions to close 2015-04-04 10:55:47,519 DEBUG [RS:0;localhost:52161] regionserver.HRegionServer(1221): {1588230740=hbase:meta,,1.1588230740} 2015-04-04 10:55:47,519 DEBUG [RS_CLOSE_META-localhost:52161-0] handler.CloseRegionHandler(119): Processing close of hbase:meta,,1.1588230740 2015-04-04 10:55:47,520 DEBUG [RS_CLOSE_META-localhost:52161-0] regionserver.HRegion(1350): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2015-04-04 10:55:47,520 DEBUG [RS_CLOSE_META-localhost:52161-0] regionserver.HRegion(1377): Updates disabled for region hbase:meta,,1.1588230740 2015-04-04 10:55:47,520 INFO [RS_CLOSE_META-localhost:52161-0] regionserver.HRegion(2084): Started memstore flush for hbase:meta,,1.1588230740, current region memstore size 21.44 KB, and 1/1 column families' memstores are being flushed. 2015-04-04 10:55:47,534 INFO [IPC Server handler 5 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741887_1063{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 0 2015-04-04 10:55:47,535 INFO [RS_CLOSE_META-localhost:52161-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=97, memsize=21.4 K, hasBloomFilter=false, into tmp file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/hbase/meta/1588230740/.tmp/011f615669674ac683a27273aa54806d 2015-04-04 10:55:47,543 INFO [RS_CLOSE_META-localhost:52161-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 011f615669674ac683a27273aa54806d 2015-04-04 10:55:47,545 DEBUG [RS_CLOSE_META-localhost:52161-0] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/hbase/meta/1588230740/.tmp/011f615669674ac683a27273aa54806d as hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/hbase/meta/1588230740/info/011f615669674ac683a27273aa54806d 2015-04-04 10:55:47,555 INFO [RS_CLOSE_META-localhost:52161-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 011f615669674ac683a27273aa54806d 2015-04-04 10:55:47,555 INFO [RS_CLOSE_META-localhost:52161-0] regionserver.HStore(973): Added hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/hbase/meta/1588230740/info/011f615669674ac683a27273aa54806d, entries=20, sequenceid=97, filesize=7.1 K 2015-04-04 10:55:47,556 INFO [RS_CLOSE_META-localhost:52161-0] regionserver.HRegion(2363): Finished memstore flush of ~21.44 KB/21952, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 36ms, sequenceid=97, compaction requested=false 2015-04-04 10:55:47,558 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(879): Closed info 2015-04-04 10:55:47,563 DEBUG [RS_CLOSE_META-localhost:52161-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:52150/user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/data/hbase/meta/1588230740/recovered.edits/100.seqid to file, newSeqId=100, maxSeqId=3 2015-04-04 10:55:47,564 DEBUG [RS_CLOSE_META-localhost:52161-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:55:47,564 DEBUG [RS_CLOSE_META-localhost:52161-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:55:47,565 DEBUG [RS_CLOSE_META-localhost:52161-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2015-04-04 10:55:47,565 INFO [RS_CLOSE_META-localhost:52161-0] regionserver.HRegion(1471): Closed hbase:meta,,1.1588230740 2015-04-04 10:55:47,565 DEBUG [RS_CLOSE_META-localhost:52161-0] handler.CloseRegionHandler(164): Closed hbase:meta,,1.1588230740 2015-04-04 10:55:47,724 INFO [RS:0;localhost:52161] regionserver.HRegionServer(1023): stopping server localhost,52161,1428170104473; all regions closed. 2015-04-04 10:55:47,724 DEBUG [RS:0;localhost:52161] wal.FSHLog(1148): Closing WAL writer in /user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/WALs/localhost,52161,1428170104473 2015-04-04 10:55:47,730 INFO [IPC Server handler 1 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741831_1007{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0e34a78e-f3d0-4444-9ae9-9772f4a64a95:NORMAL|RBW]]} size 83 2015-04-04 10:55:47,735 DEBUG [RS:0;localhost:52161] wal.FSHLog(1106): Moved 1 WAL file(s) to /user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/oldWALs 2015-04-04 10:55:47,735 INFO [RS:0;localhost:52161] wal.FSHLog(1109): Closed WAL: FSHLog localhost%2C52161%2C1428170104473..meta:.meta(num 1428170106321) 2015-04-04 10:55:47,736 DEBUG [RS:0;localhost:52161] wal.FSHLog(1148): Closing WAL writer in /user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/WALs/localhost,52161,1428170104473 2015-04-04 10:55:47,739 INFO [IPC Server handler 0 on 52150] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52151 is added to blk_1073741830_1006{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-0351fb0b-5e83-4052-a234-386f1be3edfd:NORMAL|RBW]]} size 83 2015-04-04 10:55:47,742 DEBUG [RS:0;localhost:52161] wal.FSHLog(1106): Moved 1 WAL file(s) to /user/apurtell/test-data/0cff8639-dac9-4a85-8da1-769ec5b47333/oldWALs 2015-04-04 10:55:47,742 INFO [RS:0;localhost:52161] wal.FSHLog(1109): Closed WAL: FSHLog localhost%2C52161%2C1428170104473.default:(num 1428170105380) 2015-04-04 10:55:47,743 DEBUG [RS:0;localhost:52161] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:55:47,744 DEBUG [RpcServer.reader=1,bindAddress=localhost,port=52158] ipc.RpcServer$Listener(861): RpcServer.listener,port=52158: DISCONNECTING client 127.0.0.1:52174 because read count=-1. Number of active connections: 1 2015-04-04 10:55:47,844 INFO [RS:0;localhost:52161] regionserver.Leases(146): RS:0;localhost:52161 closing leases 2015-04-04 10:55:47,844 INFO [RS:0;localhost:52161] regionserver.Leases(149): RS:0;localhost:52161 closed leases 2015-04-04 10:55:47,845 INFO [RS:0;localhost:52161] hbase.ChoreService(303): Chore service for: localhost,52161,1428170104473 had [java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@22b262e2] on shutdown 2015-04-04 10:55:47,848 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/localhost,52161,1428170104473 2015-04-04 10:55:47,848 INFO [RS:0;localhost:52161] client.ConnectionManager$HConnectionImplementation(1675): Closing zookeeper sessionid=0x14c859259d50006 2015-04-04 10:55:47,849 DEBUG [RS:0;localhost:52161] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:55:47,850 INFO [RS:0;localhost:52161] ipc.RpcServer(2193): Stopping server on 52161 2015-04-04 10:55:47,850 INFO [RpcServer.listener,port=52161] ipc.RpcServer$Listener(765): RpcServer.listener,port=52161: stopping 2015-04-04 10:55:47,851 INFO [RpcServer.responder] ipc.RpcServer$Responder(1007): RpcServer.responder: stopped 2015-04-04 10:55:47,851 INFO [RpcServer.responder] ipc.RpcServer$Responder(910): RpcServer.responder: stopping 2015-04-04 10:55:47,854 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/localhost,52161,1428170104473 2015-04-04 10:55:47,854 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/localhost,52161,1428170104473 2015-04-04 10:55:47,854 INFO [main-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [localhost,52161,1428170104473] 2015-04-04 10:55:47,854 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:52161-0x14c859259d50001, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-04-04 10:55:47,854 INFO [main-EventThread] master.ServerManager(611): Cluster shutdown set; localhost,52161,1428170104473 expired; onlineServers=0 2015-04-04 10:55:47,855 DEBUG [M:0;localhost:52158] master.HMaster(1061): Stopping service threads 2015-04-04 10:55:47,855 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-04-04 10:55:47,855 INFO [RS:0;localhost:52161] regionserver.HRegionServer(1067): stopping server localhost,52161,1428170104473; zookeeper connection closed. 2015-04-04 10:55:47,855 INFO [RS:0;localhost:52161] regionserver.HRegionServer(1070): RS:0;localhost:52161 exiting 2015-04-04 10:55:47,855 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@273ddc08] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(191): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@273ddc08 2015-04-04 10:55:47,856 INFO [main] util.JVMClusterUtil(317): Shutdown of 1 master(s) and 4 regionserver(s) complete 2015-04-04 10:55:47,856 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-04-04 10:55:47,857 INFO [M:0;localhost:52158] client.ConnectionManager$HConnectionImplementation(1675): Closing zookeeper sessionid=0x14c859259d50004 2015-04-04 10:55:47,857 DEBUG [main-EventThread] zookeeper.ZKUtil(492): master:52158-0x14c859259d50000, quorum=localhost:49474, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-04-04 10:55:47,858 DEBUG [M:0;localhost:52158] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:55:47,960 INFO [M:0;localhost:52158] hbase.ChoreService(303): Chore service for: localhost,52158,1428170104089_splitLogManager_ had [java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@406993a1] on shutdown 2015-04-04 10:55:47,960 INFO [M:0;localhost:52158] flush.MasterFlushTableProcedureManager(79): stop: server shutting down. 2015-04-04 10:55:47,961 INFO [M:0;localhost:52158] ipc.RpcServer(2193): Stopping server on 52158 2015-04-04 10:55:47,961 INFO [RpcServer.listener,port=52158] ipc.RpcServer$Listener(765): RpcServer.listener,port=52158: stopping 2015-04-04 10:55:47,962 INFO [RpcServer.responder] ipc.RpcServer$Responder(1007): RpcServer.responder: stopped 2015-04-04 10:55:47,963 INFO [RpcServer.responder] ipc.RpcServer$Responder(910): RpcServer.responder: stopping 2015-04-04 10:55:47,972 INFO [M:0;localhost:52158] zookeeper.RecoverableZooKeeper(189): Node /hbase/rs/localhost,52158,1428170104089 already deleted, retry=false 2015-04-04 10:55:47,973 INFO [M:0;localhost:52158] regionserver.HRegionServer(1067): stopping server localhost,52158,1428170104089; zookeeper connection closed. 2015-04-04 10:55:47,973 INFO [M:0;localhost:52158] regionserver.HRegionServer(1070): M:0;localhost:52158 exiting 2015-04-04 10:55:47,977 INFO [main] zookeeper.MiniZooKeeperCluster(319): Shutdown MiniZK cluster with all ZK servers 2015-04-04 10:55:47,977 WARN [main] datanode.DirectoryScanner(375): DirectoryScanner: shutdown has been called 2015-04-04 10:55:47,980 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2015-04-04 10:55:48,003 WARN [DataNode: [[[DISK]file:/Users/apurtell/src/hbase/hbase-server/target/test-data/83b500a9-e143-474d-a4ed-eb29fd09ca55/dfscluster_e0b3e3f2-5daa-4165-8420-2ac6969c1ce1/dfs/data/data1/, [DISK]file:/Users/apurtell/src/hbase/hbase-server/target/test-data/83b500a9-e143-474d-a4ed-eb29fd09ca55/dfscluster_e0b3e3f2-5daa-4165-8420-2ac6969c1ce1/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:52150] datanode.BPServiceActor(722): BPOfferService for Block pool BP-2032110495-127.0.0.1-1428170101144 (Datanode Uuid 32855a70-f6fd-4b8c-81aa-7943123ae026) service to localhost/127.0.0.1:52150 interrupted 2015-04-04 10:55:48,004 WARN [DataNode: [[[DISK]file:/Users/apurtell/src/hbase/hbase-server/target/test-data/83b500a9-e143-474d-a4ed-eb29fd09ca55/dfscluster_e0b3e3f2-5daa-4165-8420-2ac6969c1ce1/dfs/data/data1/, [DISK]file:/Users/apurtell/src/hbase/hbase-server/target/test-data/83b500a9-e143-474d-a4ed-eb29fd09ca55/dfscluster_e0b3e3f2-5daa-4165-8420-2ac6969c1ce1/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:52150] datanode.BPServiceActor(845): Ending block pool service for: Block pool BP-2032110495-127.0.0.1-1428170101144 (Datanode Uuid 32855a70-f6fd-4b8c-81aa-7943123ae026) service to localhost/127.0.0.1:52150 2015-04-04 10:55:48,013 WARN [org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor@4aea4e2e] blockmanagement.DecommissionManager$Monitor(78): Monitor interrupted: java.lang.InterruptedException: sleep interrupted 2015-04-04 10:55:48,022 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2015-04-04 10:55:48,039 INFO [main] hbase.HBaseTestingUtility(1062): Minicluster is down Tests run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 49.42 sec <<< FAILURE! - in org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface testLegacyRecovery(org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface) Time elapsed: 2.817 sec <<< FAILURE! java.lang.AssertionError: Result of org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy.getCtPreWALRestore is expected to be 1, while we get 0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.verifyMethodResult(TestRegionObserverInterface.java:751) at org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testLegacyRecovery(TestRegionObserverInterface.java:685) 2015-04-04 10:55:48,043 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@1b1325d3 2015-04-04 10:55:48,044 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-04-04 10:55:48,044 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@1b1325d3 2015-04-04 10:55:48,044 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-04-04 10:55:48,044 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@1b1325d3 2015-04-04 10:55:48,044 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-04-04 10:55:48,044 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@1b1325d3 2015-04-04 10:55:48,044 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(120): Starting fs shutdown hook thread. 2015-04-04 10:55:48,044 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. Results : Failed tests: TestRegionObserverInterface.testLegacyRecovery:685->verifyMethodResult:751 Result of org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy.getCtPreWALRestore is expected to be 1, while we get 0 Tests run: 13, Failures: 1, Errors: 0, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] HBase .............................................. SUCCESS [ 2.248 s] [INFO] HBase - Checkstyle ................................. SUCCESS [ 0.481 s] [INFO] HBase - Annotations ................................ SUCCESS [ 0.079 s] [INFO] HBase - Common ..................................... SUCCESS [ 5.370 s] [INFO] HBase - Protocol ................................... SUCCESS [ 0.100 s] [INFO] HBase - Client ..................................... SUCCESS [ 1.590 s] [INFO] HBase - Hadoop Compatibility ....................... SUCCESS [ 0.079 s] [INFO] HBase - Hadoop Two Compatibility ................... SUCCESS [ 1.182 s] [INFO] HBase - Prefix Tree ................................ SUCCESS [ 0.689 s] [INFO] HBase - Server ..................................... FAILURE [ 58.074 s] [INFO] HBase - Testing Util ............................... SKIPPED [INFO] HBase - Thrift ..................................... SKIPPED [INFO] HBase - Rest ....................................... SKIPPED [INFO] HBase - Shell ...................................... SKIPPED [INFO] HBase - Integration Tests .......................... SKIPPED [INFO] HBase - Examples ................................... SKIPPED [INFO] HBase - Assembly ................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:11 min [INFO] Finished at: 2015-04-04T10:55:48-07:00 [INFO] Final Memory: 64M/763M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.18:test (default-test) on project hbase-server: There are test failures. [ERROR] [ERROR] Please refer to /Users/apurtell/src/hbase/hbase-server/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hbase-server TEST FAILED ON RUN 6! Exiting...