2015-04-04 10:47:55,549 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testLegacyRecovery Thread=368, OpenFileDescriptor=424, MaxFileDescriptor=10240, SystemLoadAverage=229, ProcessCount=232, AvailableMemoryMB=16, ConnectionCount=1 2015-04-04 10:47:55,550 INFO [Thread-346] coprocessor.TestRegionObserverInterface(648): org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface.testLegacyRecovery 2015-04-04 10:47:55,552 INFO [B.defaultRpcServer.handler=3,queue=0,port=51327] master.HMaster(1318): Client=apurtell/null create 'TestTable.testLegacyRecovery', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:47:55,556 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testLegacyRecovery/write-master:513270000000000 2015-04-04 10:47:55,564 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(188): Create table TestTable.testLegacyRecovery 2015-04-04 10:47:55,576 INFO [IPC Server handler 1 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741851_1027{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 792 2015-04-04 10:47:55,983 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testLegacyRecovery/.tabledesc/.tableinfo.0000000001 2015-04-04 10:47:55,986 INFO [RegionOpenAndInitThread-TestTable.testLegacyRecovery-1] regionserver.HRegion(5885): creating HRegion TestTable.testLegacyRecovery HTD == 'TestTable.testLegacyRecovery', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp Table name == TestTable.testLegacyRecovery 2015-04-04 10:47:55,996 INFO [IPC Server handler 0 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741852_1028{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 63 2015-04-04 10:47:56,404 DEBUG [RegionOpenAndInitThread-TestTable.testLegacyRecovery-1] regionserver.HRegion(721): Instantiated TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,404 DEBUG [RegionOpenAndInitThread-TestTable.testLegacyRecovery-1] regionserver.HRegion(1350): Closing TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.: disabling compactions & flushes 2015-04-04 10:47:56,404 DEBUG [RegionOpenAndInitThread-TestTable.testLegacyRecovery-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,404 INFO [RegionOpenAndInitThread-TestTable.testLegacyRecovery-1] regionserver.HRegion(1471): Closed TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,410 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:47:56,411 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,51330,1428169644428 2015-04-04 10:47:56,411 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKAssign(174): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Async create of unassigned node f75016e79034bf82786de08758b9104f with OFFLINE state 2015-04-04 10:47:56,421 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:47:56,421 DEBUG [main-EventThread] master.OfflineCallback(69): rs={f75016e79034bf82786de08758b9104f state=OFFLINE, ts=1428169676411, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:47:56,422 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={f75016e79034bf82786de08758b9104f state=OFFLINE, ts=1428169676411, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:47:56,424 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1658): localhost,51330,1428169644428 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:47:56,425 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.RegionStates(1112): Transition {f75016e79034bf82786de08758b9104f state=OFFLINE, ts=1428169676411, server=null} to {f75016e79034bf82786de08758b9104f state=PENDING_OPEN, ts=1428169676425, server=localhost,51330,1428169644428} 2015-04-04 10:47:56,426 INFO [PriorityRpcServer.handler=5,queue=1,port=51330] regionserver.RSRpcServices(1390): Open TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,434 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning f75016e79034bf82786de08758b9104f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:47:56,434 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1788): Bulk assigning done for localhost,51330,1428169644428 2015-04-04 10:47:56,434 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testLegacyRecovery state from ENABLING to ENABLED 2015-04-04 10:47:56,435 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,435 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node f75016e79034bf82786de08758b9104f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:47:56,436 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(6078): Opening region: {ENCODED => f75016e79034bf82786de08758b9104f, NAME => 'TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:47:56,436 DEBUG [AM.ZK.Worker-pool3-t12] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,51330,1428169644428, region=f75016e79034bf82786de08758b9104f, current_state={f75016e79034bf82786de08758b9104f state=PENDING_OPEN, ts=1428169676425, server=localhost,51330,1428169644428} 2015-04-04 10:47:56,436 INFO [AM.ZK.Worker-pool3-t12] master.RegionStates(1112): Transition {f75016e79034bf82786de08758b9104f state=PENDING_OPEN, ts=1428169676425, server=localhost,51330,1428169644428} to {f75016e79034bf82786de08758b9104f state=OPENING, ts=1428169676436, server=localhost,51330,1428169644428} 2015-04-04 10:47:56,437 DEBUG [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:47:56,437 DEBUG [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:47:56,437 INFO [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:47:56,437 INFO [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:47:56,437 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testLegacyRecovery f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,438 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(721): Instantiated TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,441 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testLegacyRecovery/write-master:513270000000000 2015-04-04 10:47:56,442 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(218): Table, TestTable.testLegacyRecovery, creation successful 2015-04-04 10:47:56,442 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:56,442 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:47:56,443 DEBUG [StoreOpener-f75016e79034bf82786de08758b9104f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/a 2015-04-04 10:47:56,446 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:56,446 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:47:56,447 DEBUG [StoreOpener-f75016e79034bf82786de08758b9104f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/b 2015-04-04 10:47:56,449 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:56,449 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:47:56,450 DEBUG [StoreOpener-f75016e79034bf82786de08758b9104f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/c 2015-04-04 10:47:56,451 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,453 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:47:56,458 DEBUG [RS_OPEN_REGION-localhost:51330-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:47:56,458 INFO [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(847): Onlined f75016e79034bf82786de08758b9104f; next sequenceid=2 2015-04-04 10:47:56,459 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(643): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Attempting to retransition opening state of node f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,460 INFO [PostOpenDeployTasks:f75016e79034bf82786de08758b9104f] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,463 INFO [PostOpenDeployTasks:f75016e79034bf82786de08758b9104f] hbase.MetaTableAccessor(1350): Updated row TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. with server=localhost,51330,1428169644428 2015-04-04 10:47:56,464 DEBUG [PostOpenDeployTasks:f75016e79034bf82786de08758b9104f] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,465 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning f75016e79034bf82786de08758b9104f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:47:56,467 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node f75016e79034bf82786de08758b9104f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:47:56,467 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,467 DEBUG [RS_OPEN_REGION-localhost:51330-0] coordination.ZkOpenRegionCoordination(91): Transitioned f75016e79034bf82786de08758b9104f to OPENED in zk on localhost,51330,1428169644428 2015-04-04 10:47:56,467 DEBUG [RS_OPEN_REGION-localhost:51330-0] handler.OpenRegionHandler(167): Opened TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. on localhost,51330,1428169644428 2015-04-04 10:47:56,467 DEBUG [AM.ZK.Worker-pool3-t11] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,51330,1428169644428, region=f75016e79034bf82786de08758b9104f, current_state={f75016e79034bf82786de08758b9104f state=OPENING, ts=1428169676436, server=localhost,51330,1428169644428} 2015-04-04 10:47:56,468 INFO [AM.ZK.Worker-pool3-t11] master.RegionStates(1112): Transition {f75016e79034bf82786de08758b9104f state=OPENING, ts=1428169676436, server=localhost,51330,1428169644428} to {f75016e79034bf82786de08758b9104f state=OPEN, ts=1428169676468, server=localhost,51330,1428169644428} 2015-04-04 10:47:56,468 DEBUG [AM.ZK.Worker-pool3-t11] coordination.ZkOpenRegionCoordination(350): Handling OPENED of f75016e79034bf82786de08758b9104f from localhost,51327,1428169644129; deleting unassigned node 2015-04-04 10:47:56,469 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,469 DEBUG [AM.ZK.Worker-pool3-t11] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node f75016e79034bf82786de08758b9104f in expected state RS_ZK_REGION_OPENED 2015-04-04 10:47:56,469 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:47:56,469 DEBUG [AM.ZK.Worker-pool3-t11] master.AssignmentManager$4(1337): Znode TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. deleted, state: {f75016e79034bf82786de08758b9104f state=OPEN, ts=1428169676468, server=localhost,51330,1428169644428} 2015-04-04 10:47:56,469 DEBUG [AM.ZK.Worker-pool3-t11] master.RegionStates(460): Onlined f75016e79034bf82786de08758b9104f on localhost,51330,1428169644428 {ENCODED => f75016e79034bf82786de08758b9104f, NAME => 'TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:47:56,713 INFO [Thread-346] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:47:56,718 INFO [Thread-346] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:47:56,733 INFO [Thread-346] client.ConnectionUtils(103): regionserver/localhost/127.0.0.1:0 server-side HConnection retries=350 2015-04-04 10:47:56,733 INFO [Thread-346] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2015-04-04 10:47:56,736 INFO [Thread-346] ipc.RpcServer$Listener(587): regionserver/localhost/127.0.0.1:0: started 10 reader(s). 2015-04-04 10:47:56,737 INFO [Thread-346] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2015-04-04 10:47:56,739 INFO [Thread-346] zookeeper.RecoverableZooKeeper(121): Process identifier=regionserver:51420 connecting to ZooKeeper ensemble=localhost:64763 2015-04-04 10:47:56,743 DEBUG [Thread-346-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:514200x0, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-04-04 10:47:56,744 DEBUG [Thread-346] zookeeper.ZKUtil(490): regionserver:514200x0, quorum=localhost:64763, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2015-04-04 10:47:56,744 DEBUG [Thread-346] zookeeper.ZKUtil(490): regionserver:514200x0, quorum=localhost:64763, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2015-04-04 10:47:56,745 DEBUG [Thread-346-EventThread] zookeeper.ZooKeeperWatcher(485): regionserver:51420-0x14c858b55e60016 connected 2015-04-04 10:47:56,746 INFO [RpcServer.responder] ipc.RpcServer$Responder(906): RpcServer.responder: starting 2015-04-04 10:47:56,746 INFO [RpcServer.listener,port=51420] ipc.RpcServer$Listener(718): RpcServer.listener,port=51420: starting 2015-04-04 10:47:56,746 DEBUG [Thread-346] ipc.RpcExecutor(114): B.default Start Handler index=0 queue=0 2015-04-04 10:47:56,747 DEBUG [Thread-346] ipc.RpcExecutor(114): B.default Start Handler index=1 queue=0 2015-04-04 10:47:56,747 DEBUG [Thread-346] ipc.RpcExecutor(114): B.default Start Handler index=2 queue=0 2015-04-04 10:47:56,747 DEBUG [Thread-346] ipc.RpcExecutor(114): B.default Start Handler index=3 queue=0 2015-04-04 10:47:56,747 DEBUG [Thread-346] ipc.RpcExecutor(114): B.default Start Handler index=4 queue=0 2015-04-04 10:47:56,747 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=0 queue=0 2015-04-04 10:47:56,748 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=1 queue=1 2015-04-04 10:47:56,748 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=2 queue=0 2015-04-04 10:47:56,748 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=3 queue=1 2015-04-04 10:47:56,748 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=4 queue=0 2015-04-04 10:47:56,749 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=5 queue=1 2015-04-04 10:47:56,749 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=6 queue=0 2015-04-04 10:47:56,749 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=7 queue=1 2015-04-04 10:47:56,749 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=8 queue=0 2015-04-04 10:47:56,750 DEBUG [Thread-346] ipc.RpcExecutor(114): Priority Start Handler index=9 queue=1 2015-04-04 10:47:56,750 DEBUG [Thread-346] ipc.RpcExecutor(114): Replication Start Handler index=0 queue=0 2015-04-04 10:47:56,750 DEBUG [Thread-346] ipc.RpcExecutor(114): Replication Start Handler index=1 queue=0 2015-04-04 10:47:56,750 DEBUG [Thread-346] ipc.RpcExecutor(114): Replication Start Handler index=2 queue=0 2015-04-04 10:47:56,751 INFO [Thread-346] http.HttpRequestLog(69): Http request log for http.requests.regionserver is not defined 2015-04-04 10:47:56,751 INFO [Thread-346] http.HttpServer(819): Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2015-04-04 10:47:56,752 INFO [Thread-346] http.HttpServer(797): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver 2015-04-04 10:47:56,752 INFO [Thread-346] http.HttpServer(804): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2015-04-04 10:47:56,752 INFO [Thread-346] http.HttpServer(804): Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2015-04-04 10:47:56,753 INFO [Thread-346] http.HttpServer(1011): Jetty bound to port 51422 2015-04-04 10:47:56,753 INFO [Thread-346] log.Slf4jLog(67): jetty-6.1.26 2015-04-04 10:47:56,764 INFO [Thread-346] log.Slf4jLog(67): Started SelectChannelConnector@0.0.0.0:51422 2015-04-04 10:47:56,768 INFO [RS:3;localhost:51420] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x437b73e8 connecting to ZooKeeper ensemble=localhost:64763 2015-04-04 10:47:56,771 DEBUG [RS:3;localhost:51420-EventThread] zookeeper.ZooKeeperWatcher(401): hconnection-0x437b73e80x0, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-04-04 10:47:56,772 DEBUG [RS:3;localhost:51420] ipc.AbstractRpcClient(105): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@11a4bafb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-04-04 10:47:56,772 DEBUG [RS:3;localhost:51420-EventThread] zookeeper.ZooKeeperWatcher(485): hconnection-0x437b73e8-0x14c858b55e60017 connected 2015-04-04 10:47:56,774 INFO [RS:3;localhost:51420] regionserver.HRegionServer(760): ClusterId : df3fb5cb-d65f-4b00-966d-d31987a29bbe 2015-04-04 10:47:56,774 DEBUG [RS:3;localhost:51420] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2015-04-04 10:47:56,775 DEBUG [RS:3;localhost:51420] zookeeper.RecoverableZooKeeper(531): Node /hbase/online-snapshot/acquired already exists 2015-04-04 10:47:56,776 DEBUG [RS:3;localhost:51420] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2015-04-04 10:47:56,776 DEBUG [RS:3;localhost:51420] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2015-04-04 10:47:56,777 DEBUG [RS:3;localhost:51420] zookeeper.RecoverableZooKeeper(531): Node /hbase/flush-table-proc/acquired already exists 2015-04-04 10:47:56,778 DEBUG [RS:3;localhost:51420] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2015-04-04 10:47:56,779 INFO [RS:3;localhost:51420] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2015-04-04 10:47:56,779 INFO [RS:3;localhost:51420] regionserver.HRegionServer$CompactionChecker(1477): CompactionChecker runs every 1sec 2015-04-04 10:47:56,779 DEBUG [RS:3;localhost:51420] ipc.AbstractRpcClient(105): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36ae3790, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=localhost/127.0.0.1:0 2015-04-04 10:47:56,779 DEBUG [RS:3;localhost:51420] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:RS:3;localhost:51420 2015-04-04 10:47:56,780 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-04-04 10:47:56,780 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-04-04 10:47:56,781 DEBUG [RS:3;localhost:51420] zookeeper.ZKUtil(490): regionserver:51420-0x14c858b55e60016, quorum=localhost:64763, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,51420,1428169676736 2015-04-04 10:47:56,781 INFO [RS:3;localhost:51420] regionserver.RegionServerCoprocessorHost(65): System coprocessor loading is enabled 2015-04-04 10:47:56,781 INFO [RS:3;localhost:51420] regionserver.RegionServerCoprocessorHost(66): Table coprocessor loading is enabled 2015-04-04 10:47:56,781 DEBUG [main-EventThread] zookeeper.ZKUtil(490): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,51330,1428169644428 2015-04-04 10:47:56,781 DEBUG [main-EventThread] zookeeper.ZKUtil(490): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,51330,1428169644428 2015-04-04 10:47:56,782 INFO [RS:3;localhost:51420] regionserver.HRegionServer(2196): reportForDuty to master=localhost,51327,1428169644129 with port=51420, startcode=1428169676736 2015-04-04 10:47:56,782 DEBUG [RS:3;localhost:51420] ipc.RpcClientImpl$Connection(343): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2015-04-04 10:47:56,782 DEBUG [main-EventThread] zookeeper.ZKUtil(490): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,51420,1428169676736 2015-04-04 10:47:56,782 DEBUG [main-EventThread] zookeeper.ZKUtil(490): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,51420,1428169676736 2015-04-04 10:47:56,782 DEBUG [RS:3;localhost:51420] ipc.RpcClientImpl$Connection(708): Connecting to localhost/127.0.0.1:51327 2015-04-04 10:47:56,783 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/localhost,51330,1428169644428 2015-04-04 10:47:56,783 DEBUG [RpcServer.listener,port=51327] ipc.RpcServer$Listener(829): RpcServer.listener,port=51327: connection from 127.0.0.1:51424; # active connections: 3 2015-04-04 10:47:56,784 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/localhost,51420,1428169676736 2015-04-04 10:47:56,784 INFO [RpcServer.reader=5,bindAddress=localhost,port=51327] ipc.RpcServer$Connection(1661): Connection from 127.0.0.1 port: 51424 with version info: version: "1.1.0-SNAPSHOT" url: "git://apurtell-ltm2.internal.salesforce.com/Users/apurtell/src/hbase" revision: "5c19f9eac7d1f22147345f09aad2f04fec1ad67d" user: "apurtell" date: "Sat Apr 4 10:46:42 PDT 2015" src_checksum: "2a921de567e8e6d9e1c358f3276c1ec9" 2015-04-04 10:47:56,785 INFO [B.defaultRpcServer.handler=0,queue=0,port=51327] master.ServerManager(448): Registering server=localhost,51420,1428169676736 2015-04-04 10:47:56,786 INFO [RS:3;localhost:51420] regionserver.HRegionServer(1315): Config from master: hbase.rootdir=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7 2015-04-04 10:47:56,786 INFO [RS:3;localhost:51420] regionserver.HRegionServer(1315): Config from master: fs.defaultFS=hdfs://localhost:51319 2015-04-04 10:47:56,786 INFO [RS:3;localhost:51420] regionserver.HRegionServer(1315): Config from master: hbase.master.info.port=51329 2015-04-04 10:47:56,786 WARN [RS:3;localhost:51420] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2015-04-04 10:47:56,786 INFO [RS:3;localhost:51420] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:56,787 DEBUG [RS:3;localhost:51420] regionserver.HRegionServer(1574): logdir=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/WALs/localhost,51420,1428169676736 2015-04-04 10:47:56,794 DEBUG [RS:3;localhost:51420] regionserver.Replication(143): ReplicationStatisticsThread 300 2015-04-04 10:47:56,794 INFO [RS:3;localhost:51420] wal.WALFactory(144): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2015-04-04 10:47:56,800 INFO [RS:3;localhost:51420] wal.FSHLog(550): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=localhost%2C51420%2C1428169676736.default, suffix=, logDir=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/WALs/localhost,51420,1428169676736, archiveDir=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/oldWALs 2015-04-04 10:47:56,809 INFO [RS:3;localhost:51420] wal.FSHLog(1495): Slow sync cost: 6 ms, current pipeline: [] 2015-04-04 10:47:56,809 INFO [RS:3;localhost:51420] wal.FSHLog(972): New WAL /user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/WALs/localhost,51420,1428169676736/localhost%2C51420%2C1428169676736.default.1428169676800 2015-04-04 10:47:56,810 INFO [RS:3;localhost:51420] regionserver.MetricsRegionServerWrapperImpl(100): Computing regionserver metrics every 5000 milliseconds 2015-04-04 10:47:56,811 DEBUG [RS:3;localhost:51420] executor.ExecutorService(99): Starting executor service name=RS_OPEN_REGION-localhost:51420, corePoolSize=3, maxPoolSize=3 2015-04-04 10:47:56,811 DEBUG [RS:3;localhost:51420] executor.ExecutorService(99): Starting executor service name=RS_OPEN_META-localhost:51420, corePoolSize=1, maxPoolSize=1 2015-04-04 10:47:56,811 DEBUG [RS:3;localhost:51420] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_REGION-localhost:51420, corePoolSize=3, maxPoolSize=3 2015-04-04 10:47:56,811 DEBUG [RS:3;localhost:51420] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_META-localhost:51420, corePoolSize=1, maxPoolSize=1 2015-04-04 10:47:56,812 DEBUG [RS:3;localhost:51420] executor.ExecutorService(99): Starting executor service name=RS_LOG_REPLAY_OPS-localhost:51420, corePoolSize=2, maxPoolSize=2 2015-04-04 10:47:56,812 DEBUG [RS:3;localhost:51420] executor.ExecutorService(99): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-localhost:51420, corePoolSize=3, maxPoolSize=3 2015-04-04 10:47:56,816 DEBUG [RS:3;localhost:51420] zookeeper.ZKUtil(490): regionserver:51420-0x14c858b55e60016, quorum=localhost:64763, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,51330,1428169644428 2015-04-04 10:47:56,817 DEBUG [RS:3;localhost:51420] zookeeper.ZKUtil(490): regionserver:51420-0x14c858b55e60016, quorum=localhost:64763, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,51420,1428169676736 2015-04-04 10:47:56,817 INFO [RS:3;localhost:51420] regionserver.ReplicationSourceManager(227): Current list of replicators: [localhost,51330,1428169644428, localhost,51399,1428169671757, localhost,51420,1428169676736, localhost,51371,1428169649535] other RSs: [localhost,51330,1428169644428, localhost,51420,1428169676736] 2015-04-04 10:47:56,867 INFO [RS:3;localhost:51420] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x377da43b connecting to ZooKeeper ensemble=localhost:64763 2015-04-04 10:47:56,870 DEBUG [RS:3;localhost:51420-EventThread] zookeeper.ZooKeeperWatcher(401): hconnection-0x377da43b0x0, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-04-04 10:47:56,871 DEBUG [RS:3;localhost:51420] ipc.AbstractRpcClient(105): Codec=org.apache.hadoop.hbase.codec.KeyValueCodecWithTags@9766fa6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-04-04 10:47:56,872 DEBUG [RS:3;localhost:51420-EventThread] zookeeper.ZooKeeperWatcher(485): hconnection-0x377da43b-0x14c858b55e60018 connected 2015-04-04 10:47:56,892 INFO [RS:3;localhost:51420] regionserver.HeapMemoryManager(187): Starting HeapMemoryTuner chore. 2015-04-04 10:47:56,892 INFO [SplitLogWorker-localhost:51420] regionserver.SplitLogWorker(134): SplitLogWorker localhost,51420,1428169676736 starting 2015-04-04 10:47:56,893 INFO [RS:3;localhost:51420] regionserver.HRegionServer(1337): Serving as localhost,51420,1428169676736, RpcServer on localhost/127.0.0.1:51420, sessionid=0x14c858b55e60016 2015-04-04 10:47:56,893 DEBUG [RS:3;localhost:51420] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2015-04-04 10:47:56,893 DEBUG [RS:3;localhost:51420] snapshot.RegionServerSnapshotManager(122): Start Snapshot Manager localhost,51420,1428169676736 2015-04-04 10:47:56,893 DEBUG [RS:3;localhost:51420] procedure.ZKProcedureMemberRpcs(349): Starting procedure member 'localhost,51420,1428169676736' 2015-04-04 10:47:56,893 DEBUG [RS:3;localhost:51420] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2015-04-04 10:47:56,894 DEBUG [RS:3;localhost:51420] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2015-04-04 10:47:56,895 DEBUG [RS:3;localhost:51420] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2015-04-04 10:47:56,895 DEBUG [RS:3;localhost:51420] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2015-04-04 10:47:56,895 DEBUG [RS:3;localhost:51420] flush.RegionServerFlushTableProcedureManager(101): Start region server flush procedure manager localhost,51420,1428169676736 2015-04-04 10:47:56,895 DEBUG [RS:3;localhost:51420] procedure.ZKProcedureMemberRpcs(349): Starting procedure member 'localhost,51420,1428169676736' 2015-04-04 10:47:56,895 DEBUG [RS:3;localhost:51420] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2015-04-04 10:47:56,896 DEBUG [RS:3;localhost:51420] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-04-04 10:47:56,896 DEBUG [RS:3;localhost:51420] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2015-04-04 10:47:56,899 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51327] master.ServerManager(951): New admin connection to localhost,51420,1428169676736 2015-04-04 10:47:56,899 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51327] ipc.RpcClientImpl$Connection(343): Use SIMPLE authentication for service AdminService, sasl=false 2015-04-04 10:47:56,899 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51327] ipc.RpcClientImpl$Connection(708): Connecting to localhost/127.0.0.1:51420 2015-04-04 10:47:56,901 DEBUG [RpcServer.listener,port=51420] ipc.RpcServer$Listener(829): RpcServer.listener,port=51420: connection from 127.0.0.1:51428; # active connections: 1 2015-04-04 10:47:56,902 INFO [RpcServer.reader=1,bindAddress=localhost,port=51420] ipc.RpcServer$Connection(1661): Connection from 127.0.0.1 port: 51428 with version info: version: "1.1.0-SNAPSHOT" url: "git://apurtell-ltm2.internal.salesforce.com/Users/apurtell/src/hbase" revision: "5c19f9eac7d1f22147345f09aad2f04fec1ad67d" user: "apurtell" date: "Sat Apr 4 10:46:42 PDT 2015" src_checksum: "2a921de567e8e6d9e1c358f3276c1ec9" 2015-04-04 10:47:56,903 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51420] regionserver.RSRpcServices(1498): Warming up Region TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,908 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51420] regionserver.HRegion(6141): HRegion.Warming up region: {ENCODED => f75016e79034bf82786de08758b9104f, NAME => 'TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:47:56,909 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51420] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:47:56,909 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51420] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:47:56,909 INFO [B.defaultRpcServer.handler=0,queue=0,port=51420] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:47:56,909 INFO [B.defaultRpcServer.handler=0,queue=0,port=51420] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:47:56,909 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51420] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testLegacyRecovery f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,909 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51420] regionserver.HRegion(721): Instantiated TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,912 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:56,912 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:47:56,913 DEBUG [StoreOpener-f75016e79034bf82786de08758b9104f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/a 2015-04-04 10:47:56,915 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:56,915 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:47:56,916 DEBUG [StoreOpener-f75016e79034bf82786de08758b9104f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/b 2015-04-04 10:47:56,917 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:56,917 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:47:56,919 DEBUG [StoreOpener-f75016e79034bf82786de08758b9104f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/c 2015-04-04 10:47:56,919 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51420] regionserver.HRegion(1350): Closing TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.: disabling compactions & flushes 2015-04-04 10:47:56,919 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51420] regionserver.HRegion(1377): Updates disabled for region TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,919 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.-1] regionserver.HStore(879): Closed a 2015-04-04 10:47:56,920 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.-1] regionserver.HStore(879): Closed b 2015-04-04 10:47:56,920 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.-1] regionserver.HStore(879): Closed c 2015-04-04 10:47:56,924 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51420] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:47:56,924 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51420] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:47:56,925 INFO [B.defaultRpcServer.handler=0,queue=0,port=51420] regionserver.HRegion(1471): Closed TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,925 INFO [B.defaultRpcServer.handler=3,queue=0,port=51327] master.HMaster(1289): Client=apurtell/null move hri=TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f., src=localhost,51330,1428169644428, dest=localhost,51420,1428169676736, running balancer 2015-04-04 10:47:56,925 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51327] master.AssignmentManager(2452): Starting unassign of TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. (offlining), current state: {f75016e79034bf82786de08758b9104f state=OPEN, ts=1428169676469, server=localhost,51330,1428169644428} 2015-04-04 10:47:56,925 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51327] zookeeper.ZKAssign(527): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Creating unassigned node f75016e79034bf82786de08758b9104f in a CLOSING state 2015-04-04 10:47:56,926 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:47:56,927 INFO [B.defaultRpcServer.handler=3,queue=0,port=51327] master.RegionStates(1112): Transition {f75016e79034bf82786de08758b9104f state=OPEN, ts=1428169676469, server=localhost,51330,1428169644428} to {f75016e79034bf82786de08758b9104f state=PENDING_CLOSE, ts=1428169676927, server=localhost,51330,1428169644428} 2015-04-04 10:47:56,927 INFO [PriorityRpcServer.handler=2,queue=0,port=51330] regionserver.RSRpcServices(1018): Close f75016e79034bf82786de08758b9104f, moving to localhost,51420,1428169676736 2015-04-04 10:47:56,928 DEBUG [RS_CLOSE_REGION-localhost:51330-0] handler.CloseRegionHandler(119): Processing close of TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,928 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51327] master.AssignmentManager(1836): Sent CLOSE to localhost,51330,1428169644428 for region TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,929 DEBUG [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(1350): Closing TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.: disabling compactions & flushes 2015-04-04 10:47:56,929 DEBUG [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(1377): Updates disabled for region TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,929 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.-1] regionserver.HStore(879): Closed a 2015-04-04 10:47:56,930 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.-1] regionserver.HStore(879): Closed b 2015-04-04 10:47:56,930 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.-1] regionserver.HStore(879): Closed c 2015-04-04 10:47:56,936 DEBUG [RS_CLOSE_REGION-localhost:51330-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/recovered.edits/4.seqid to file, newSeqId=4, maxSeqId=2 2015-04-04 10:47:56,937 DEBUG [RS_CLOSE_REGION-localhost:51330-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:47:56,937 DEBUG [RS_CLOSE_REGION-localhost:51330-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:47:56,938 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(1471): Closed TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,938 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegionServer(2951): Adding moved region record: f75016e79034bf82786de08758b9104f to localhost,51420,1428169676736 as of 2 2015-04-04 10:47:56,938 DEBUG [RS_CLOSE_REGION-localhost:51330-0] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning f75016e79034bf82786de08758b9104f from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:47:56,939 DEBUG [RS_CLOSE_REGION-localhost:51330-0] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node f75016e79034bf82786de08758b9104f from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:47:56,939 DEBUG [RS_CLOSE_REGION-localhost:51330-0] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. on localhost,51330,1428169644428 2015-04-04 10:47:56,939 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,940 DEBUG [RS_CLOSE_REGION-localhost:51330-0] handler.CloseRegionHandler(164): Closed TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,940 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,51330,1428169644428, region=f75016e79034bf82786de08758b9104f, current_state={f75016e79034bf82786de08758b9104f state=PENDING_CLOSE, ts=1428169676927, server=localhost,51330,1428169644428} 2015-04-04 10:47:56,940 DEBUG [AM.ZK.Worker-pool3-t14] handler.ClosedRegionHandler(93): Handling CLOSED event for f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,940 INFO [AM.ZK.Worker-pool3-t14] master.RegionStates(1112): Transition {f75016e79034bf82786de08758b9104f state=PENDING_CLOSE, ts=1428169676927, server=localhost,51330,1428169644428} to {f75016e79034bf82786de08758b9104f state=CLOSED, ts=1428169676940, server=localhost,51330,1428169644428} 2015-04-04 10:47:56,941 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(2363): Found an existing plan for TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. destination server is localhost,51420,1428169676736 accepted as a dest server = true 2015-04-04 10:47:56,941 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(2401): Using pre-existing plan for TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.; plan=hri=TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f., src=localhost,51330,1428169644428, dest=localhost,51420,1428169676736 2015-04-04 10:47:56,941 INFO [AM.ZK.Worker-pool3-t14] master.RegionStates(1112): Transition {f75016e79034bf82786de08758b9104f state=CLOSED, ts=1428169676940, server=localhost,51330,1428169644428} to {f75016e79034bf82786de08758b9104f state=OFFLINE, ts=1428169676941, server=localhost,51330,1428169644428} 2015-04-04 10:47:56,941 DEBUG [AM.ZK.Worker-pool3-t14] zookeeper.ZKAssign(205): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Creating (or updating) unassigned node f75016e79034bf82786de08758b9104f with OFFLINE state 2015-04-04 10:47:56,942 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,942 INFO [AM.ZK.Worker-pool3-t14] master.AssignmentManager(2104): Assigning TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. to localhost,51420,1428169676736 2015-04-04 10:47:56,943 INFO [AM.ZK.Worker-pool3-t14] master.RegionStates(1112): Transition {f75016e79034bf82786de08758b9104f state=OFFLINE, ts=1428169676941, server=localhost,51330,1428169644428} to {f75016e79034bf82786de08758b9104f state=PENDING_OPEN, ts=1428169676943, server=localhost,51420,1428169676736} 2015-04-04 10:47:56,944 INFO [PriorityRpcServer.handler=0,queue=0,port=51420] regionserver.RSRpcServices(1390): Open TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,949 DEBUG [RS_OPEN_REGION-localhost:51420-0] zookeeper.ZKAssign(805): regionserver:51420-0x14c858b55e60016, quorum=localhost:64763, baseZNode=/hbase Transitioning f75016e79034bf82786de08758b9104f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:47:56,950 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(948): Handling M_ZK_REGION_OFFLINE, server=localhost,51420,1428169676736, region=f75016e79034bf82786de08758b9104f, current_state={f75016e79034bf82786de08758b9104f state=PENDING_OPEN, ts=1428169676943, server=localhost,51420,1428169676736} 2015-04-04 10:47:56,951 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,951 DEBUG [RS_OPEN_REGION-localhost:51420-0] zookeeper.ZKAssign(880): regionserver:51420-0x14c858b55e60016, quorum=localhost:64763, baseZNode=/hbase Transitioned node f75016e79034bf82786de08758b9104f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:47:56,951 DEBUG [RS_OPEN_REGION-localhost:51420-0] regionserver.HRegion(6078): Opening region: {ENCODED => f75016e79034bf82786de08758b9104f, NAME => 'TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:47:56,951 DEBUG [AM.ZK.Worker-pool3-t17] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,51420,1428169676736, region=f75016e79034bf82786de08758b9104f, current_state={f75016e79034bf82786de08758b9104f state=PENDING_OPEN, ts=1428169676943, server=localhost,51420,1428169676736} 2015-04-04 10:47:56,952 INFO [AM.ZK.Worker-pool3-t17] master.RegionStates(1112): Transition {f75016e79034bf82786de08758b9104f state=PENDING_OPEN, ts=1428169676943, server=localhost,51420,1428169676736} to {f75016e79034bf82786de08758b9104f state=OPENING, ts=1428169676952, server=localhost,51420,1428169676736} 2015-04-04 10:47:56,952 DEBUG [RS_OPEN_REGION-localhost:51420-0] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:47:56,952 DEBUG [RS_OPEN_REGION-localhost:51420-0] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:47:56,952 INFO [RS_OPEN_REGION-localhost:51420-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:47:56,952 INFO [RS_OPEN_REGION-localhost:51420-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:47:56,952 DEBUG [RS_OPEN_REGION-localhost:51420-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testLegacyRecovery f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,953 DEBUG [RS_OPEN_REGION-localhost:51420-0] regionserver.HRegion(721): Instantiated TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,955 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:56,956 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:47:56,957 DEBUG [StoreOpener-f75016e79034bf82786de08758b9104f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/a 2015-04-04 10:47:56,958 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:56,958 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:47:56,959 DEBUG [StoreOpener-f75016e79034bf82786de08758b9104f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/b 2015-04-04 10:47:56,961 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:56,961 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:47:56,962 DEBUG [StoreOpener-f75016e79034bf82786de08758b9104f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/c 2015-04-04 10:47:56,965 DEBUG [RS_OPEN_REGION-localhost:51420-0] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,967 DEBUG [RS_OPEN_REGION-localhost:51420-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:47:56,973 DEBUG [RS_OPEN_REGION-localhost:51420-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/recovered.edits/5.seqid to file, newSeqId=5, maxSeqId=4 2015-04-04 10:47:56,974 INFO [RS_OPEN_REGION-localhost:51420-0] regionserver.HRegion(847): Onlined f75016e79034bf82786de08758b9104f; next sequenceid=5 2015-04-04 10:47:56,975 DEBUG [RS_OPEN_REGION-localhost:51420-0] zookeeper.ZKAssign(643): regionserver:51420-0x14c858b55e60016, quorum=localhost:64763, baseZNode=/hbase Attempting to retransition opening state of node f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,976 INFO [PostOpenDeployTasks:f75016e79034bf82786de08758b9104f] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,979 DEBUG [hconnection-0x437b73e8-shared--pool66-t1] ipc.RpcClientImpl$Connection(343): Use SIMPLE authentication for service ClientService, sasl=false 2015-04-04 10:47:56,979 DEBUG [hconnection-0x437b73e8-shared--pool66-t1] ipc.RpcClientImpl$Connection(708): Connecting to localhost/127.0.0.1:51330 2015-04-04 10:47:56,980 DEBUG [RpcServer.listener,port=51330] ipc.RpcServer$Listener(829): RpcServer.listener,port=51330: connection from 127.0.0.1:51429; # active connections: 6 2015-04-04 10:47:56,980 INFO [RpcServer.reader=7,bindAddress=localhost,port=51330] ipc.RpcServer$Connection(1661): Connection from 127.0.0.1 port: 51429 with version info: version: "1.1.0-SNAPSHOT" url: "git://apurtell-ltm2.internal.salesforce.com/Users/apurtell/src/hbase" revision: "5c19f9eac7d1f22147345f09aad2f04fec1ad67d" user: "apurtell" date: "Sat Apr 4 10:46:42 PDT 2015" src_checksum: "2a921de567e8e6d9e1c358f3276c1ec9" 2015-04-04 10:47:56,982 INFO [PostOpenDeployTasks:f75016e79034bf82786de08758b9104f] hbase.MetaTableAccessor(1350): Updated row TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. with server=localhost,51420,1428169676736 2015-04-04 10:47:56,983 DEBUG [PostOpenDeployTasks:f75016e79034bf82786de08758b9104f] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:56,983 DEBUG [RS_OPEN_REGION-localhost:51420-0] zookeeper.ZKAssign(805): regionserver:51420-0x14c858b55e60016, quorum=localhost:64763, baseZNode=/hbase Transitioning f75016e79034bf82786de08758b9104f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:47:56,984 DEBUG [RS_OPEN_REGION-localhost:51420-0] zookeeper.ZKAssign(880): regionserver:51420-0x14c858b55e60016, quorum=localhost:64763, baseZNode=/hbase Transitioned node f75016e79034bf82786de08758b9104f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:47:56,985 DEBUG [RS_OPEN_REGION-localhost:51420-0] coordination.ZkOpenRegionCoordination(91): Transitioned f75016e79034bf82786de08758b9104f to OPENED in zk on localhost,51420,1428169676736 2015-04-04 10:47:56,984 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,985 DEBUG [RS_OPEN_REGION-localhost:51420-0] handler.OpenRegionHandler(167): Opened TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. on localhost,51420,1428169676736 2015-04-04 10:47:56,986 DEBUG [AM.ZK.Worker-pool3-t18] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,51420,1428169676736, region=f75016e79034bf82786de08758b9104f, current_state={f75016e79034bf82786de08758b9104f state=OPENING, ts=1428169676952, server=localhost,51420,1428169676736} 2015-04-04 10:47:56,986 INFO [AM.ZK.Worker-pool3-t18] master.RegionStates(1112): Transition {f75016e79034bf82786de08758b9104f state=OPENING, ts=1428169676952, server=localhost,51420,1428169676736} to {f75016e79034bf82786de08758b9104f state=OPEN, ts=1428169676986, server=localhost,51420,1428169676736} 2015-04-04 10:47:56,986 DEBUG [AM.ZK.Worker-pool3-t18] coordination.ZkOpenRegionCoordination(350): Handling OPENED of f75016e79034bf82786de08758b9104f from localhost,51327,1428169644129; deleting unassigned node 2015-04-04 10:47:56,987 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:56,987 DEBUG [AM.ZK.Worker-pool3-t18] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node f75016e79034bf82786de08758b9104f in expected state RS_ZK_REGION_OPENED 2015-04-04 10:47:56,988 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:47:56,988 DEBUG [AM.ZK.Worker-pool3-t18] master.AssignmentManager$4(1337): Znode TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. deleted, state: {f75016e79034bf82786de08758b9104f state=OPEN, ts=1428169676986, server=localhost,51420,1428169676736} 2015-04-04 10:47:56,988 DEBUG [AM.ZK.Worker-pool3-t18] master.RegionStates(460): Onlined f75016e79034bf82786de08758b9104f on localhost,51420,1428169676736 {ENCODED => f75016e79034bf82786de08758b9104f, NAME => 'TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:47:56,988 INFO [AM.ZK.Worker-pool3-t18] master.RegionStates(472): Offlined f75016e79034bf82786de08758b9104f from localhost,51330,1428169644428 2015-04-04 10:47:57,044 DEBUG [hconnection-0x3b4ad720-shared--pool16-t40] ipc.RpcClientImpl$Connection(343): Use SIMPLE authentication for service ClientService, sasl=false 2015-04-04 10:47:57,044 DEBUG [hconnection-0x3b4ad720-shared--pool16-t40] ipc.RpcClientImpl$Connection(708): Connecting to localhost/127.0.0.1:51420 2015-04-04 10:47:57,045 DEBUG [RpcServer.listener,port=51420] ipc.RpcServer$Listener(829): RpcServer.listener,port=51420: connection from 127.0.0.1:51430; # active connections: 2 2015-04-04 10:47:57,046 INFO [RpcServer.reader=2,bindAddress=localhost,port=51420] ipc.RpcServer$Connection(1661): Connection from 127.0.0.1 port: 51430 with version info: version: "1.1.0-SNAPSHOT" url: "git://apurtell-ltm2.internal.salesforce.com/Users/apurtell/src/hbase" revision: "5c19f9eac7d1f22147345f09aad2f04fec1ad67d" user: "apurtell" date: "Sat Apr 4 10:46:42 PDT 2015" src_checksum: "2a921de567e8e6d9e1c358f3276c1ec9" 2015-04-04 10:47:57,050 INFO [Thread-346] hbase.MiniHBaseCluster(245): Killing localhost,51420,1428169676736 2015-04-04 10:47:57,050 FATAL [Thread-346] regionserver.HRegionServer(1982): ABORTING region server localhost,51420,1428169676736: Simulated kill 2015-04-04 10:47:57,051 FATAL [Thread-346] regionserver.HRegionServer(1988): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy, org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver, org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2015-04-04 10:47:57,054 INFO [Thread-346] regionserver.HRegionServer(1992): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "HeapMemoryUsage" : { "committed" : 344457216, "init" : 134217728, "max" : 2609905664, "used" : 133616616 }, "ObjectPendingFinalizationCount" : 0, "NonHeapMemoryUsage" : { "committed" : 60227584, "init" : 24576000, "max" : 318767104, "used" : 60062968 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "localhost", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : 6, "numActiveHandler" : 0, "TotalCallTime_num_ops" : 341, "TotalCallTime_min" : 0, "TotalCallTime_max" : 26, "TotalCallTime_mean" : 1.4222873900293256, "TotalCallTime_median" : 1.0, "TotalCallTime_75th_percentile" : 1.0, "TotalCallTime_95th_percentile" : 5.0, "TotalCallTime_99th_percentile" : 23.579999999999984, "QueueCallTime_num_ops" : 341, "QueueCallTime_min" : 0, "QueueCallTime_max" : 6, "QueueCallTime_mean" : 0.10557184750733138, "QueueCallTime_median" : 0.0, "QueueCallTime_75th_percentile" : 0.0, "QueueCallTime_95th_percentile" : 1.0, "QueueCallTime_99th_percentile" : 1.0, "authenticationFailures" : 0, "authorizationFailures" : 0, "authenticationSuccesses" : 0, "authorizationSuccesses" : 23, "ProcessCallTime_num_ops" : 341, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 26, "ProcessCallTime_mean" : 1.316715542521994, "ProcessCallTime_median" : 1.0, "ProcessCallTime_75th_percentile" : 1.0, "ProcessCallTime_95th_percentile" : 5.0, "ProcessCallTime_99th_percentile" : 23.579999999999984, "sentBytes" : 82799, "receivedBytes" : 31828 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "localhost", "sink.appliedOps" : 0, "sink.appliedBatches" : 0, "sink.ageOfLastAppliedOp" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:64763", "tag.serverName" : "localhost,51330,1428169644428", "tag.clusterId" : "df3fb5cb-d65f-4b00-966d-d31987a29bbe", "tag.Context" : "regionserver", "tag.Hostname" : "localhost", "regionCount" : 2, "storeCount" : 2, "hlogFileCount" : 2, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 10360, "storeFileSize" : 0, "regionServerStartTime" : 1428169644428, "totalRequestCount" : 347, "readRequestCount" : 138, "writeRequestCount" : 25, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0, "percentFilesLocalSecondaryRegions" : 0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1042890544, "blockCacheCount" : 0, "blockCacheSize" : 1071760, "blockCacheHitCount" : 0, "blockCacheMissCount" : 0, "blockCacheEvictionCount" : 0, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "blockedRequestCount" : 0, "splitSuccessCounnt" : 0, "splitRequestCount" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0.0, "Append_median" : 0.0, "Append_75th_percentile" : 0.0, "Append_95th_percentile" : 0.0, "Append_99th_percentile" : 0.0, "Delete_num_ops" : 4, "Delete_min" : 1, "Delete_max" : 2, "Delete_mean" : 1.25, "Delete_median" : 1.0, "Delete_75th_percentile" : 1.75, "Delete_95th_percentile" : 2.0, "Delete_99th_percentile" : 2.0, "Mutate_num_ops" : 21, "Mutate_min" : 1, "Mutate_max" : 18, "Mutate_mean" : 2.4761904761904763, "Mutate_median" : 2.0, "Mutate_75th_percentile" : 2.0, "Mutate_95th_percentile" : 16.399999999999977, "Mutate_99th_percentile" : 18.0, "ScanNext_num_ops" : 99, "ScanNext_min" : 0, "ScanNext_max" : 1072, "ScanNext_mean" : 770.4545454545455, "ScanNext_median" : 1017.0, "ScanNext_75th_percentile" : 1042.0, "ScanNext_95th_percentile" : 1072.0, "ScanNext_99th_percentile" : 1072.0, "slowDeleteCount" : 0, "slowIncrementCount" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0.0, "FlushTime_median" : 0.0, "FlushTime_75th_percentile" : 0.0, "FlushTime_95th_percentile" : 0.0, "FlushTime_99th_percentile" : 0.0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 2, "Get_mean" : 0.75, "Get_median" : 0.5, "Get_75th_percentile" : 1.75, "Get_95th_percentile" : 2.0, "Get_99th_percentile" : 2.0, "Replay_num_ops" : 2, "Replay_min" : 2, "Replay_max" : 2, "Replay_mean" : 2.0, "Replay_median" : 2.0, "Replay_75th_percentile" : 2.0, "Replay_95th_percentile" : 2.0, "Replay_99th_percentile" : 2.0, "slowGetCount" : 0, "slowAppendCount" : 0, "slowPutCount" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0.0, "SplitTime_median" : 0.0, "SplitTime_75th_percentile" : 0.0, "SplitTime_95th_percentile" : 0.0, "SplitTime_99th_percentile" : 0.0, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0.0, "Increment_median" : 0.0, "Increment_75th_percentile" : 0.0, "Increment_95th_percentile" : 0.0, "Increment_99th_percentile" : 0.0 } ] } 2015-04-04 10:47:57,055 ERROR [B.defaultRpcServer.handler=4,queue=0,port=51327] master.MasterRpcServices(328): Region server localhost,51420,1428169676736 reported a fatal error: ABORTING region server localhost,51420,1428169676736: Simulated kill 2015-04-04 10:47:57,056 INFO [Thread-346] regionserver.HRegionServer(1823): STOPPED: Simulated kill 2015-04-04 10:47:57,056 INFO [RS:3;localhost:51420] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-04-04 10:47:57,056 INFO [RS:3;localhost:51420] regionserver.HRegionServer(955): Stopping infoServer 2015-04-04 10:47:57,056 INFO [SplitLogWorker-localhost:51420] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-04-04 10:47:57,056 INFO [SplitLogWorker-localhost:51420] regionserver.SplitLogWorker(155): SplitLogWorker localhost,51420,1428169676736 exiting 2015-04-04 10:47:57,061 INFO [RS:3;localhost:51420] log.Slf4jLog(67): Stopped SelectChannelConnector@0.0.0.0:0 2015-04-04 10:47:57,062 INFO [RS:3;localhost:51420] regionserver.HeapMemoryManager(198): Stoping HeapMemoryTuner chore. 2015-04-04 10:47:57,062 INFO [RS:3;localhost:51420] snapshot.RegionServerSnapshotManager(134): Stopping RegionServerSnapshotManager abruptly. 2015-04-04 10:47:57,062 INFO [RS:3;localhost:51420] flush.RegionServerFlushTableProcedureManager(113): Stopping region server flush procedure manager abruptly. 2015-04-04 10:47:57,063 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-04-04 10:47:57,062 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-04-04 10:47:57,063 DEBUG [RS:3;localhost:51420] zookeeper.MetaTableLocator(619): Stopping MetaTableLocator 2015-04-04 10:47:57,063 INFO [RS:3;localhost:51420] client.ConnectionManager$HConnectionImplementation(1675): Closing zookeeper sessionid=0x14c858b55e60017 2015-04-04 10:47:57,064 DEBUG [RS:3;localhost:51420] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:47:57,065 DEBUG [RpcServer.reader=7,bindAddress=localhost,port=51330] ipc.RpcServer$Listener(861): RpcServer.listener,port=51330: DISCONNECTING client 127.0.0.1:51429 because read count=-1. Number of active connections: 6 2015-04-04 10:47:57,166 DEBUG [RS:3;localhost:51420] wal.FSHLog(1148): Closing WAL writer in /user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/WALs/localhost,51420,1428169676736 2015-04-04 10:47:57,171 INFO [IPC Server handler 6 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741853_1029{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 928 2015-04-04 10:47:57,574 DEBUG [RS:3;localhost:51420] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:47:57,575 DEBUG [RpcServer.reader=5,bindAddress=localhost,port=51327] ipc.RpcServer$Listener(861): RpcServer.listener,port=51327: DISCONNECTING client 127.0.0.1:51424 because read count=-1. Number of active connections: 3 2015-04-04 10:47:57,680 INFO [RS:3;localhost:51420] regionserver.Leases(146): RS:3;localhost:51420 closing leases 2015-04-04 10:47:57,680 INFO [RS:3;localhost:51420] regionserver.Leases(149): RS:3;localhost:51420 closed leases 2015-04-04 10:47:57,681 INFO [RS:3;localhost:51420] ipc.RpcServer(2193): Stopping server on 51420 2015-04-04 10:47:57,681 INFO [RpcServer.listener,port=51420] ipc.RpcServer$Listener(765): RpcServer.listener,port=51420: stopping 2015-04-04 10:47:57,682 INFO [RpcServer.responder] ipc.RpcServer$Responder(1007): RpcServer.responder: stopped 2015-04-04 10:47:57,682 INFO [RpcServer.responder] ipc.RpcServer$Responder(910): RpcServer.responder: stopping 2015-04-04 10:47:57,693 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/localhost,51420,1428169676736 2015-04-04 10:47:57,693 DEBUG [Thread-346-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51420-0x14c858b55e60016, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/localhost,51420,1428169676736 2015-04-04 10:47:57,693 DEBUG [Thread-346-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51420-0x14c858b55e60016, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-04-04 10:47:57,693 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/localhost,51420,1428169676736 2015-04-04 10:47:57,693 INFO [main-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [localhost,51420,1428169676736] 2015-04-04 10:47:57,695 INFO [RS:3;localhost:51420] regionserver.HRegionServer(1067): stopping server localhost,51420,1428169676736; zookeeper connection closed. 2015-04-04 10:47:57,695 INFO [RS:3;localhost:51420] regionserver.HRegionServer(1070): RS:3;localhost:51420 exiting 2015-04-04 10:47:57,695 DEBUG [main-EventThread] zookeeper.ZKUtil(490): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,51330,1428169644428 2015-04-04 10:47:57,695 INFO [main-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(121): /hbase/rs/localhost,51420,1428169676736 znode expired, triggering replicatorRemoved event 2015-04-04 10:47:57,696 DEBUG [main-EventThread] master.AssignmentManager(3327): based on AM, current region=hbase:meta,,1.1588230740 is on server=localhost,51330,1428169644428 server being checked: localhost,51420,1428169676736 2015-04-04 10:47:57,696 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-04-04 10:47:57,696 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@70c6cb3f] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(191): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@70c6cb3f 2015-04-04 10:47:57,697 DEBUG [main-EventThread] master.ServerManager(627): Added=localhost,51420,1428169676736 to dead servers, submitted shutdown handler to be executed meta=false 2015-04-04 10:47:57,697 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-04-04 10:47:57,698 DEBUG [main-EventThread] zookeeper.ZKUtil(490): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,51330,1428169644428 2015-04-04 10:47:57,699 DEBUG [main-EventThread] zookeeper.ZKUtil(490): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/localhost,51330,1428169644428 2015-04-04 10:47:57,700 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/localhost,51330,1428169644428 2015-04-04 10:47:57,705 INFO [MASTER_SERVER_OPERATIONS-localhost:51327-2] handler.ServerShutdownHandler(206): Mark regions in recovery for crashed server localhost,51420,1428169676736 before assignment; regions=[{ENCODED => f75016e79034bf82786de08758b9104f, NAME => 'TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:47:57,706 DEBUG [MASTER_SERVER_OPERATIONS-localhost:51327-2] zookeeper.ZKUtil(749): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Unable to get data of znode /hbase/recovering-regions/f75016e79034bf82786de08758b9104f because node does not exist (not an error) 2015-04-04 10:47:57,708 DEBUG [MASTER_SERVER_OPERATIONS-localhost:51327-2] coordination.ZKSplitLogManagerCoordination(650): Marked f75016e79034bf82786de08758b9104f as recovering from localhost,51420,1428169676736: /hbase/recovering-regions/f75016e79034bf82786de08758b9104f/localhost,51420,1428169676736 2015-04-04 10:47:57,708 DEBUG [MASTER_SERVER_OPERATIONS-localhost:51327-2] master.RegionStates(532): Adding to processed servers localhost,51420,1428169676736 2015-04-04 10:47:57,709 INFO [MASTER_SERVER_OPERATIONS-localhost:51327-2] master.RegionStates(1112): Transition {f75016e79034bf82786de08758b9104f state=OPEN, ts=1428169676988, server=localhost,51420,1428169676736} to {f75016e79034bf82786de08758b9104f state=OFFLINE, ts=1428169677709, server=localhost,51420,1428169676736} 2015-04-04 10:47:57,709 INFO [MASTER_SERVER_OPERATIONS-localhost:51327-2] handler.ServerShutdownHandler(238): Reassigning 1 region(s) that localhost,51420,1428169676736 was carrying (and 0 regions(s) that were opening on this server) 2015-04-04 10:47:57,709 INFO [MASTER_SERVER_OPERATIONS-localhost:51327-2] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,51330,1428169644428 2015-04-04 10:47:57,709 DEBUG [MASTER_SERVER_OPERATIONS-localhost:51327-2] zookeeper.ZKAssign(174): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Async create of unassigned node f75016e79034bf82786de08758b9104f with OFFLINE state 2015-04-04 10:47:57,710 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:47:57,710 DEBUG [main-EventThread] master.OfflineCallback(69): rs={f75016e79034bf82786de08758b9104f state=OFFLINE, ts=1428169677709, server=localhost,51420,1428169676736}, server=localhost,51330,1428169644428 2015-04-04 10:47:57,711 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={f75016e79034bf82786de08758b9104f state=OFFLINE, ts=1428169677709, server=localhost,51420,1428169676736}, server=localhost,51330,1428169644428 2015-04-04 10:47:57,714 DEBUG [MASTER_SERVER_OPERATIONS-localhost:51327-2] master.AssignmentManager(1658): localhost,51330,1428169644428 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:47:57,715 INFO [MASTER_SERVER_OPERATIONS-localhost:51327-2] master.RegionStates(1112): Transition {f75016e79034bf82786de08758b9104f state=OFFLINE, ts=1428169677709, server=localhost,51420,1428169676736} to {f75016e79034bf82786de08758b9104f state=PENDING_OPEN, ts=1428169677715, server=localhost,51330,1428169644428} 2015-04-04 10:47:57,716 INFO [PriorityRpcServer.handler=6,queue=0,port=51330] regionserver.RSRpcServices(1390): Open TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:57,723 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning f75016e79034bf82786de08758b9104f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:47:57,723 DEBUG [MASTER_SERVER_OPERATIONS-localhost:51327-2] master.AssignmentManager(1788): Bulk assigning done for localhost,51330,1428169644428 2015-04-04 10:47:57,723 INFO [MASTER_SERVER_OPERATIONS-localhost:51327-2] master.AssignmentManager(3252): Waiting for f75016e79034bf82786de08758b9104f to leave regions-in-transition, timeOut=15000 ms. 2015-04-04 10:47:57,724 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node f75016e79034bf82786de08758b9104f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:47:57,724 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:57,725 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(6078): Opening region: {ENCODED => f75016e79034bf82786de08758b9104f, NAME => 'TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:47:57,725 DEBUG [AM.ZK.Worker-pool3-t2] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,51330,1428169644428, region=f75016e79034bf82786de08758b9104f, current_state={f75016e79034bf82786de08758b9104f state=PENDING_OPEN, ts=1428169677715, server=localhost,51330,1428169644428} 2015-04-04 10:47:57,725 INFO [AM.ZK.Worker-pool3-t2] master.RegionStates(1112): Transition {f75016e79034bf82786de08758b9104f state=PENDING_OPEN, ts=1428169677715, server=localhost,51330,1428169644428} to {f75016e79034bf82786de08758b9104f state=OPENING, ts=1428169677725, server=localhost,51330,1428169644428} 2015-04-04 10:47:57,726 DEBUG [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:47:57,726 DEBUG [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:47:57,726 INFO [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:47:57,726 INFO [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:47:57,726 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testLegacyRecovery f75016e79034bf82786de08758b9104f 2015-04-04 10:47:57,727 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(721): Instantiated TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:57,729 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:57,730 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:47:57,731 DEBUG [StoreOpener-f75016e79034bf82786de08758b9104f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/a 2015-04-04 10:47:57,732 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:57,732 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:47:57,733 DEBUG [StoreOpener-f75016e79034bf82786de08758b9104f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/b 2015-04-04 10:47:57,734 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:57,735 INFO [StoreOpener-f75016e79034bf82786de08758b9104f-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:47:57,736 DEBUG [StoreOpener-f75016e79034bf82786de08758b9104f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/c 2015-04-04 10:47:57,739 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:57,741 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:47:57,746 DEBUG [RS_OPEN_REGION-localhost:51330-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/recovered.edits/40000005.seqid to file, newSeqId=40000005, maxSeqId=5 2015-04-04 10:47:57,747 INFO [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(847): Onlined f75016e79034bf82786de08758b9104f; next sequenceid=40000005 2015-04-04 10:47:57,747 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(643): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Attempting to retransition opening state of node f75016e79034bf82786de08758b9104f 2015-04-04 10:47:57,748 INFO [PostOpenDeployTasks:f75016e79034bf82786de08758b9104f] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:57,750 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/recovering-regions/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:57,751 DEBUG [PostOpenDeployTasks:f75016e79034bf82786de08758b9104f] regionserver.HRegionServer(3103): Update last flushed sequence id of region f75016e79034bf82786de08758b9104f for localhost,51420,1428169676736 2015-04-04 10:47:57,754 INFO [PostOpenDeployTasks:f75016e79034bf82786de08758b9104f] hbase.MetaTableAccessor(1350): Updated row TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. with server=localhost,51330,1428169644428 2015-04-04 10:47:57,754 DEBUG [PostOpenDeployTasks:f75016e79034bf82786de08758b9104f] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:57,755 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning f75016e79034bf82786de08758b9104f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:47:57,756 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:57,756 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node f75016e79034bf82786de08758b9104f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:47:57,756 DEBUG [RS_OPEN_REGION-localhost:51330-1] coordination.ZkOpenRegionCoordination(91): Transitioned f75016e79034bf82786de08758b9104f to OPENED in zk on localhost,51330,1428169644428 2015-04-04 10:47:57,757 DEBUG [RS_OPEN_REGION-localhost:51330-1] handler.OpenRegionHandler(167): Opened TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. on localhost,51330,1428169644428 2015-04-04 10:47:57,757 DEBUG [AM.ZK.Worker-pool3-t20] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,51330,1428169644428, region=f75016e79034bf82786de08758b9104f, current_state={f75016e79034bf82786de08758b9104f state=OPENING, ts=1428169677725, server=localhost,51330,1428169644428} 2015-04-04 10:47:57,757 INFO [AM.ZK.Worker-pool3-t20] master.RegionStates(1112): Transition {f75016e79034bf82786de08758b9104f state=OPENING, ts=1428169677725, server=localhost,51330,1428169644428} to {f75016e79034bf82786de08758b9104f state=OPEN, ts=1428169677757, server=localhost,51330,1428169644428} 2015-04-04 10:47:57,757 DEBUG [AM.ZK.Worker-pool3-t20] coordination.ZkOpenRegionCoordination(350): Handling OPENED of f75016e79034bf82786de08758b9104f from localhost,51327,1428169644129; deleting unassigned node 2015-04-04 10:47:57,759 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:57,759 DEBUG [AM.ZK.Worker-pool3-t20] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node f75016e79034bf82786de08758b9104f in expected state RS_ZK_REGION_OPENED 2015-04-04 10:47:57,759 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:47:57,759 DEBUG [AM.ZK.Worker-pool3-t20] master.AssignmentManager$4(1337): Znode TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. deleted, state: {f75016e79034bf82786de08758b9104f state=OPEN, ts=1428169677757, server=localhost,51330,1428169644428} 2015-04-04 10:47:57,759 DEBUG [AM.ZK.Worker-pool3-t20] master.RegionStates(460): Onlined f75016e79034bf82786de08758b9104f on localhost,51330,1428169644428 {ENCODED => f75016e79034bf82786de08758b9104f, NAME => 'TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:47:57,759 INFO [AM.ZK.Worker-pool3-t20] master.RegionStates(472): Offlined f75016e79034bf82786de08758b9104f from localhost,51420,1428169676736 2015-04-04 10:47:57,760 DEBUG [MASTER_SERVER_OPERATIONS-localhost:51327-2] master.DeadServer(118): Finished processing localhost,51420,1428169676736 2015-04-04 10:47:57,762 DEBUG [M_LOG_REPLAY_OPS-localhost:51327-2] master.MasterFileSystem(330): Renamed region directory: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/WALs/localhost,51420,1428169676736-splitting 2015-04-04 10:47:57,762 INFO [M_LOG_REPLAY_OPS-localhost:51327-2] master.SplitLogManager(560): dead splitlog workers [localhost,51420,1428169676736] 2015-04-04 10:47:57,764 DEBUG [M_LOG_REPLAY_OPS-localhost:51327-2] master.SplitLogManager(254): Scheduling batch of logs to split 2015-04-04 10:47:57,764 INFO [M_LOG_REPLAY_OPS-localhost:51327-2] master.SplitLogManager(256): started splitting 1 logs in [hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/WALs/localhost,51420,1428169676736-splitting] for [localhost,51420,1428169676736] 2015-04-04 10:47:57,765 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2015-04-04 10:47:57,765 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination(432): put up splitlog task at znode /hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800 2015-04-04 10:47:57,766 DEBUG [main-EventThread] coordination.ZkSplitLogWorkerCoordination(107): tasks arrived or departed 2015-04-04 10:47:57,766 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination(465): task not yet acquired /hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800 ver = 0 2015-04-04 10:47:57,768 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800 2015-04-04 10:47:57,768 INFO [SplitLogWorker-localhost:51330] coordination.ZkSplitLogWorkerCoordination(263): worker localhost,51330,1428169644428 acquired task /hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800 2015-04-04 10:47:57,769 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(560): task /hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800 acquired by localhost,51330,1428169644428 2015-04-04 10:47:57,785 INFO [RS_LOG_REPLAY_OPS-localhost:51330-0] wal.WALSplitter(289): Splitting wal: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/WALs/localhost,51420,1428169676736-splitting/localhost%2C51420%2C1428169676736.default.1428169676800, length=928 2015-04-04 10:47:57,785 INFO [RS_LOG_REPLAY_OPS-localhost:51330-0] wal.WALSplitter(290): DistributedLogReplay = true 2015-04-04 10:47:57,786 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800 2015-04-04 10:47:57,786 INFO [RS_LOG_REPLAY_OPS-localhost:51330-0] util.FSHDFSUtils(173): Recovering lease on dfs file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/WALs/localhost,51420,1428169676736-splitting/localhost%2C51420%2C1428169676736.default.1428169676800 2015-04-04 10:47:57,786 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800 2015-04-04 10:47:57,788 INFO [RS_LOG_REPLAY_OPS-localhost:51330-0] util.FSHDFSUtils(260): recoverLease=true, attempt=0 on file=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/WALs/localhost,51420,1428169676736-splitting/localhost%2C51420%2C1428169676736.default.1428169676800 after 1ms 2015-04-04 10:47:57,795 DEBUG [RS_LOG_REPLAY_OPS-localhost:51330-0-Writer-0] wal.WALSplitter$WriterThread(1074): Writer thread Thread[RS_LOG_REPLAY_OPS-localhost:51330-0-Writer-0,5,main]: starting 2015-04-04 10:47:57,795 DEBUG [RS_LOG_REPLAY_OPS-localhost:51330-0-Writer-1] wal.WALSplitter$WriterThread(1074): Writer thread Thread[RS_LOG_REPLAY_OPS-localhost:51330-0-Writer-1,5,main]: starting 2015-04-04 10:47:57,795 DEBUG [RS_LOG_REPLAY_OPS-localhost:51330-0-Writer-2] wal.WALSplitter$WriterThread(1074): Writer thread Thread[RS_LOG_REPLAY_OPS-localhost:51330-0-Writer-2,5,main]: starting 2015-04-04 10:47:57,798 DEBUG [RS_LOG_REPLAY_OPS-localhost:51330-0] wal.WALSplitter(385): Finishing writing output logs and closing down. 2015-04-04 10:47:57,798 DEBUG [RS_LOG_REPLAY_OPS-localhost:51330-0] wal.WALSplitter$OutputSink(1205): Waiting for split writer threads to finish 2015-04-04 10:47:57,799 INFO [RS_LOG_REPLAY_OPS-localhost:51330-0-Writer-2] zookeeper.RecoverableZooKeeper(121): Process identifier=hconnection-0x3c073f4b connecting to ZooKeeper ensemble=localhost:64763 2015-04-04 10:47:57,802 DEBUG [RS_LOG_REPLAY_OPS-localhost:51330-0-Writer-2-EventThread] zookeeper.ZooKeeperWatcher(401): hconnection-0x3c073f4b0x0, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-04-04 10:47:57,803 DEBUG [RS_LOG_REPLAY_OPS-localhost:51330-0-Writer-2] ipc.AbstractRpcClient(105): Codec=org.apache.hadoop.hbase.regionserver.wal.WALCellCodec@148a8965, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2015-04-04 10:47:57,803 DEBUG [RS_LOG_REPLAY_OPS-localhost:51330-0-Writer-2-EventThread] zookeeper.ZooKeeperWatcher(485): hconnection-0x3c073f4b-0x14c858b55e60019 connected 2015-04-04 10:47:57,805 DEBUG [hconnection-0x3c073f4b-metaLookup-shared--pool68-t1] ipc.RpcClientImpl$Connection(343): Use SIMPLE authentication for service ClientService, sasl=false 2015-04-04 10:47:57,806 DEBUG [hconnection-0x3c073f4b-metaLookup-shared--pool68-t1] ipc.RpcClientImpl$Connection(708): Connecting to localhost/127.0.0.1:51330 2015-04-04 10:47:57,806 DEBUG [RpcServer.listener,port=51330] ipc.RpcServer$Listener(829): RpcServer.listener,port=51330: connection from 127.0.0.1:51432; # active connections: 6 2015-04-04 10:47:57,807 INFO [RpcServer.reader=8,bindAddress=localhost,port=51330] ipc.RpcServer$Connection(1661): Connection from 127.0.0.1 port: 51432 with version info: version: "1.1.0-SNAPSHOT" url: "git://apurtell-ltm2.internal.salesforce.com/Users/apurtell/src/hbase" revision: "5c19f9eac7d1f22147345f09aad2f04fec1ad67d" user: "apurtell" date: "Sat Apr 4 10:46:42 PDT 2015" src_checksum: "2a921de567e8e6d9e1c358f3276c1ec9" 2015-04-04 10:47:57,810 DEBUG [RS_LOG_REPLAY_OPS-localhost:51330-0-Writer-2] ipc.RpcClientImpl$Connection(343): Use SIMPLE authentication for service AdminService, sasl=false 2015-04-04 10:47:57,810 DEBUG [RS_LOG_REPLAY_OPS-localhost:51330-0-Writer-2] ipc.RpcClientImpl$Connection(708): Connecting to localhost/127.0.0.1:51330 2015-04-04 10:47:57,810 DEBUG [RpcServer.listener,port=51330] ipc.RpcServer$Listener(829): RpcServer.listener,port=51330: connection from 127.0.0.1:51433; # active connections: 7 2015-04-04 10:47:57,811 INFO [RpcServer.reader=9,bindAddress=localhost,port=51330] ipc.RpcServer$Connection(1661): Connection from 127.0.0.1 port: 51433 with version info: version: "1.1.0-SNAPSHOT" url: "git://apurtell-ltm2.internal.salesforce.com/Users/apurtell/src/hbase" revision: "5c19f9eac7d1f22147345f09aad2f04fec1ad67d" user: "apurtell" date: "Sat Apr 4 10:46:42 PDT 2015" src_checksum: "2a921de567e8e6d9e1c358f3276c1ec9" 2015-04-04 10:47:57,815 INFO [regionserver/localhost/127.0.0.1:0.logRoller] regionserver.LogRoller(163): LogRoller exiting. 2015-04-04 10:47:57,815 INFO [localhost,51420,1428169676736_ChoreService_1] hbase.ScheduledChore(179): Chore: localhost,51420,1428169676736-MemstoreFlusherChore was stopped 2015-04-04 10:47:57,816 INFO [regionserver/localhost/127.0.0.1:0.leaseChecker] regionserver.Leases(146): regionserver/localhost/127.0.0.1:0.leaseChecker closing leases 2015-04-04 10:47:57,816 DEBUG [RS_LOG_REPLAY_OPS-localhost:51330-0-Writer-2] wal.WALEditsReplaySink(144): number of rows:1 are sent by batch! spent 3(ms)! 2015-04-04 10:47:57,816 INFO [regionserver/localhost/127.0.0.1:0.leaseChecker] regionserver.Leases(149): regionserver/localhost/127.0.0.1:0.leaseChecker closed leases 2015-04-04 10:47:58,058 INFO [Thread-346] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:47:58,065 INFO [Thread-346] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:47:58,066 INFO [Thread-346] coprocessor.TestRegionObserverInterface(683): All regions assigned 2015-04-04 10:47:58,067 INFO [Thread-346] client.HBaseAdmin$9(979): Started disable of TestTable.testLegacyRecovery 2015-04-04 10:47:58,067 INFO [B.defaultRpcServer.handler=2,queue=0,port=51327] master.HMaster(1659): Client=apurtell/null disable TestTable.testLegacyRecovery 2015-04-04 10:47:58,078 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testLegacyRecovery/write-master:513270000000001 2015-04-04 10:47:58,085 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(129): Attempting to disable table TestTable.testLegacyRecovery 2015-04-04 10:47:58,086 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testLegacyRecovery state from DISABLING to DISABLING 2015-04-04 10:47:58,087 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:47:58,088 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. (offlining), current state: {f75016e79034bf82786de08758b9104f state=OPEN, ts=1428169677759, server=localhost,51330,1428169644428} 2015-04-04 10:47:58,088 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Creating unassigned node f75016e79034bf82786de08758b9104f in a CLOSING state 2015-04-04 10:47:58,089 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:47:58,089 INFO [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {f75016e79034bf82786de08758b9104f state=OPEN, ts=1428169677759, server=localhost,51330,1428169644428} to {f75016e79034bf82786de08758b9104f state=PENDING_CLOSE, ts=1428169678089, server=localhost,51330,1428169644428} 2015-04-04 10:47:58,090 INFO [PriorityRpcServer.handler=1,queue=1,port=51330] regionserver.RSRpcServices(1018): Close f75016e79034bf82786de08758b9104f, moving to null 2015-04-04 10:47:58,090 DEBUG [RS_CLOSE_REGION-localhost:51330-1] handler.CloseRegionHandler(119): Processing close of TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:58,090 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,51330,1428169644428 for region TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:58,091 DEBUG [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(1350): Closing TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.: disabling compactions & flushes 2015-04-04 10:47:58,091 DEBUG [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:58,091 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(2084): Started memstore flush for TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f., current region memstore size 480 B, and 3/3 column families' memstores are being flushed. 2015-04-04 10:47:58,092 DEBUG [Thread-346] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in TestTable.testLegacyRecovery 2015-04-04 10:47:58,102 INFO [IPC Server handler 5 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741854_1030{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 0 2015-04-04 10:47:58,103 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=40000007, memsize=160, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/.tmp/7ffa4201646c4f59a1d57435b46e14b5 2015-04-04 10:47:58,121 INFO [IPC Server handler 7 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741855_1031{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 4727 2015-04-04 10:47:58,199 DEBUG [Thread-346] client.HBaseAdmin(1019): Sleeping= 200ms, waiting for all regions to be disabled in TestTable.testLegacyRecovery 2015-04-04 10:47:58,285 INFO [localhost,51327,1428169644129_splitLogManager__ChoreService_1] master.SplitLogManager$TimeoutMonitor(745): total tasks = 1 unassigned = 0 tasks={/hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800=last_update = 1428169677787 last_version = 2 cur_worker_name = localhost,51330,1428169644428 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2015-04-04 10:47:58,321 INFO [RS_LOG_REPLAY_OPS-localhost:51330-0] wal.WALSplitter$OutputSink(1229): Split writers finished 2015-04-04 10:47:58,323 INFO [RS_LOG_REPLAY_OPS-localhost:51330-0] client.ConnectionManager$HConnectionImplementation(1675): Closing zookeeper sessionid=0x14c858b55e60019 2015-04-04 10:47:58,325 DEBUG [RS_LOG_REPLAY_OPS-localhost:51330-0] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:47:58,325 DEBUG [RpcServer.reader=9,bindAddress=localhost,port=51330] ipc.RpcServer$Listener(861): RpcServer.listener,port=51330: DISCONNECTING client 127.0.0.1:51433 because read count=-1. Number of active connections: 7 2015-04-04 10:47:58,325 DEBUG [RpcServer.reader=8,bindAddress=localhost,port=51330] ipc.RpcServer$Listener(861): RpcServer.listener,port=51330: DISCONNECTING client 127.0.0.1:51432 because read count=-1. Number of active connections: 7 2015-04-04 10:47:58,411 DEBUG [Thread-346] client.HBaseAdmin(1019): Sleeping= 300ms, waiting for all regions to be disabled in TestTable.testLegacyRecovery 2015-04-04 10:47:58,425 INFO [RS_LOG_REPLAY_OPS-localhost:51330-0] wal.WALSplitter(407): Processed 2 edits across 1 regions; edits skipped=1; log file=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/WALs/localhost,51420,1428169676736-splitting/localhost%2C51420%2C1428169676736.default.1428169676800, length=928, corrupted=false, progress failed=false 2015-04-04 10:47:58,433 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800 2015-04-04 10:47:58,433 INFO [RS_LOG_REPLAY_OPS-localhost:51330-0] coordination.ZkSplitLogWorkerCoordination(596): successfully transitioned task /hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800 to final state DONE localhost,51330,1428169644428 2015-04-04 10:47:58,433 INFO [RS_LOG_REPLAY_OPS-localhost:51330-0] handler.WALSplitterHandler(101): worker localhost,51330,1428169644428 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@69f71c75 in 665ms 2015-04-04 10:47:58,433 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800 2015-04-04 10:47:58,434 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(473): task /hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800 entered state: DONE localhost,51330,1428169644428 2015-04-04 10:47:58,440 INFO [main-EventThread] wal.WALSplitter(497): Archived processed log hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/WALs/localhost,51420,1428169676736-splitting/localhost%2C51420%2C1428169676736.default.1428169676800 to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/oldWALs/localhost%2C51420%2C1428169676736.default.1428169676800 2015-04-04 10:47:58,441 INFO [main-EventThread] coordination.ZKSplitLogManagerCoordination(516): Done splitting /hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800 2015-04-04 10:47:58,442 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800 2015-04-04 10:47:58,442 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2015-04-04 10:47:58,442 DEBUG [main-EventThread] coordination.ZkSplitLogWorkerCoordination(107): tasks arrived or departed 2015-04-04 10:47:58,442 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800 2015-04-04 10:47:58,442 DEBUG [main-EventThread] coordination.ZKSplitLogManagerCoordination$DeleteAsyncCallback(1078): deleted /hbase/splitWAL/WALs%2Flocalhost%2C51420%2C1428169676736-splitting%2Flocalhost%252C51420%252C1428169676736.default.1428169676800 2015-04-04 10:47:58,442 DEBUG [M_LOG_REPLAY_OPS-localhost:51327-2] coordination.ZKSplitLogManagerCoordination(324): Processing recovering [f75016e79034bf82786de08758b9104f] and servers [localhost,51420,1428169676736], isMetaRecovery=false 2015-04-04 10:47:58,444 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/f75016e79034bf82786de08758b9104f/localhost,51420,1428169676736 2015-04-04 10:47:58,445 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:58,445 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/recovering-regions/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:58,446 INFO [RS_OPEN_REGION-localhost:51330-2] handler.FinishRegionRecoveringHandler(52): /hbase/recovering-regions/f75016e79034bf82786de08758b9104f deleted; f75016e79034bf82786de08758b9104f recovered. 2015-04-04 10:47:58,447 INFO [M_LOG_REPLAY_OPS-localhost:51327-2] master.SplitLogManager(315): finished splitting (more than or equal to) 928 bytes in 1 log files in [hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/WALs/localhost,51420,1428169676736-splitting] in 683ms 2015-04-04 10:47:58,447 DEBUG [M_LOG_REPLAY_OPS-localhost:51327-2] master.DeadServer(118): Finished processing localhost,51420,1428169676736 2015-04-04 10:47:58,447 INFO [M_LOG_REPLAY_OPS-localhost:51327-2] handler.LogReplayHandler(86): Finished processing shutdown of localhost,51420,1428169676736 2015-04-04 10:47:58,525 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=40000007, memsize=160, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/.tmp/6d77b731ff814e7b837e6dfc14d3a783 2015-04-04 10:47:58,546 INFO [IPC Server handler 7 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741856_1032{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 4727 2015-04-04 10:47:58,723 DEBUG [Thread-346] client.HBaseAdmin(1019): Sleeping= 500ms, waiting for all regions to be disabled in TestTable.testLegacyRecovery 2015-04-04 10:47:58,950 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=40000007, memsize=160, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/.tmp/303e338c50d444109a32a02f4c4a4abe 2015-04-04 10:47:58,961 DEBUG [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/.tmp/7ffa4201646c4f59a1d57435b46e14b5 as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/a/7ffa4201646c4f59a1d57435b46e14b5 2015-04-04 10:47:58,969 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/a/7ffa4201646c4f59a1d57435b46e14b5, entries=1, sequenceid=40000007, filesize=4.6 K 2015-04-04 10:47:58,971 DEBUG [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/.tmp/6d77b731ff814e7b837e6dfc14d3a783 as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/b/6d77b731ff814e7b837e6dfc14d3a783 2015-04-04 10:47:58,980 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/b/6d77b731ff814e7b837e6dfc14d3a783, entries=1, sequenceid=40000007, filesize=4.6 K 2015-04-04 10:47:58,982 DEBUG [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/.tmp/303e338c50d444109a32a02f4c4a4abe as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/c/303e338c50d444109a32a02f4c4a4abe 2015-04-04 10:47:58,990 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/c/303e338c50d444109a32a02f4c4a4abe, entries=1, sequenceid=40000007, filesize=4.6 K 2015-04-04 10:47:58,991 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(2363): Finished memstore flush of ~480 B/480, currentsize=0 B/0 for region TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. in 900ms, sequenceid=40000007, compaction requested=false 2015-04-04 10:47:58,993 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.-1] regionserver.HStore(879): Closed a 2015-04-04 10:47:58,995 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.-1] regionserver.HStore(879): Closed b 2015-04-04 10:47:58,996 INFO [StoreCloserThread-TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.-1] regionserver.HStore(879): Closed c 2015-04-04 10:47:59,004 DEBUG [RS_CLOSE_REGION-localhost:51330-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/recovered.edits/40000012.seqid to file, newSeqId=40000012, maxSeqId=40000005 2015-04-04 10:47:59,005 DEBUG [RS_CLOSE_REGION-localhost:51330-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:47:59,005 DEBUG [RS_CLOSE_REGION-localhost:51330-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:47:59,005 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(1471): Closed TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:59,005 DEBUG [RS_CLOSE_REGION-localhost:51330-1] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning f75016e79034bf82786de08758b9104f from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:47:59,015 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:59,015 DEBUG [RS_CLOSE_REGION-localhost:51330-1] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node f75016e79034bf82786de08758b9104f from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:47:59,015 DEBUG [RS_CLOSE_REGION-localhost:51330-1] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. on localhost,51330,1428169644428 2015-04-04 10:47:59,015 DEBUG [RS_CLOSE_REGION-localhost:51330-1] handler.CloseRegionHandler(164): Closed TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:59,016 DEBUG [AM.ZK.Worker-pool3-t8] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,51330,1428169644428, region=f75016e79034bf82786de08758b9104f, current_state={f75016e79034bf82786de08758b9104f state=PENDING_CLOSE, ts=1428169678089, server=localhost,51330,1428169644428} 2015-04-04 10:47:59,016 DEBUG [AM.ZK.Worker-pool3-t8] handler.ClosedRegionHandler(93): Handling CLOSED event for f75016e79034bf82786de08758b9104f 2015-04-04 10:47:59,016 DEBUG [AM.ZK.Worker-pool3-t8] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. 2015-04-04 10:47:59,018 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:59,018 DEBUG [AM.ZK.Worker-pool3-t8] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node f75016e79034bf82786de08758b9104f in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:47:59,018 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:47:59,018 INFO [AM.ZK.Worker-pool3-t8] master.RegionStates(1112): Transition {f75016e79034bf82786de08758b9104f state=PENDING_CLOSE, ts=1428169678089, server=localhost,51330,1428169644428} to {f75016e79034bf82786de08758b9104f state=OFFLINE, ts=1428169679018, server=localhost,51330,1428169644428} 2015-04-04 10:47:59,019 INFO [AM.ZK.Worker-pool3-t8] master.RegionStates(600): Offlined f75016e79034bf82786de08758b9104f from localhost,51330,1428169644428 2015-04-04 10:47:59,040 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testLegacyRecovery state from DISABLING to DISABLED 2015-04-04 10:47:59,042 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(191): Disabled table, TestTable.testLegacyRecovery, is done=true 2015-04-04 10:47:59,043 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testLegacyRecovery/write-master:513270000000001 2015-04-04 10:47:59,235 INFO [Thread-346] client.HBaseAdmin(1034): Disabled TestTable.testLegacyRecovery 2015-04-04 10:47:59,236 INFO [B.defaultRpcServer.handler=0,queue=0,port=51327] master.HMaster(1564): Client=apurtell/null delete TestTable.testLegacyRecovery 2015-04-04 10:47:59,247 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testLegacyRecovery/write-master:513270000000002 2015-04-04 10:47:59,253 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table TestTable.testLegacyRecovery 2015-04-04 10:47:59,258 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:47:59,261 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => f75016e79034bf82786de08758b9104f, NAME => 'TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:47:59,264 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(186): Archiving region TestTable.testLegacyRecovery,,1428169675552.f75016e79034bf82786de08758b9104f. from FS 2015-04-04 10:47:59,264 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:59,270 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/recovered.edits] 2015-04-04 10:47:59,279 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/a/7ffa4201646c4f59a1d57435b46e14b5, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/a/7ffa4201646c4f59a1d57435b46e14b5 2015-04-04 10:47:59,287 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/b/6d77b731ff814e7b837e6dfc14d3a783, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/b/6d77b731ff814e7b837e6dfc14d3a783 2015-04-04 10:47:59,295 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/c/303e338c50d444109a32a02f4c4a4abe, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/c/303e338c50d444109a32a02f4c4a4abe 2015-04-04 10:47:59,302 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/recovered.edits/40000012.seqid, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f/recovered.edits/40000012.seqid 2015-04-04 10:47:59,303 INFO [IPC Server handler 2 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741852_1028 127.0.0.1:51320 2015-04-04 10:47:59,304 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testLegacyRecovery/f75016e79034bf82786de08758b9104f 2015-04-04 10:47:59,305 INFO [IPC Server handler 4 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741851_1027 127.0.0.1:51320 2015-04-04 10:47:59,306 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(196): Table 'TestTable.testLegacyRecovery' archived! 2015-04-04 10:47:59,306 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(116): Removing 'TestTable.testLegacyRecovery' descriptor. 2015-04-04 10:47:59,307 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(122): Removing 'TestTable.testLegacyRecovery' from region states. 2015-04-04 10:47:59,307 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(126): Marking 'TestTable.testLegacyRecovery' as deleted. 2015-04-04 10:47:59,312 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testLegacyRecovery/write-master:513270000000002 2015-04-04 10:47:59,371 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51327] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = TestTable.testLegacyRecovery org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testLegacyRecovery at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:47:59,372 INFO [Thread-346] client.HBaseAdmin(739): Deleted TestTable.testLegacyRecovery 2015-04-04 10:47:59,409 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testLegacyRecovery Thread=409 (was 368) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t40 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS_OPEN_REGION-localhost:51420-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t74 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51391 [Waiting for operation #33] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51416 [Waiting for operation #9] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t42 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t77 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t73 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS:3;localhost:51420-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t75 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t49 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t43 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t39 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: IPC Client (1121977622) connection to localhost/127.0.0.1:51420 from apurtell java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.waitForWork(RpcClientImpl.java:542) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:557) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t46 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t48 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x34c6ce8e-shared--pool9-t9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: IPC Client (1121977622) connection to localhost/127.0.0.1:51319 from apurtell.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:904) org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) Potentially hanging thread: LeaseRenewer:apurtell.hfs.3@localhost:51319 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t72 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t47 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t69 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t44 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t71 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t45 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t79 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t38 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t76 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51417 [Waiting for operation #8] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: localhost:51420Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS:3;localhost:51420-SendThread(localhost:64763) sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t68 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t67 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t70 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: MASTER_SERVER_OPERATIONS-localhost:51327-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t78 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: IPC Client (1121977622) connection to localhost/127.0.0.1:51420 from apurtell java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.waitForWork(RpcClientImpl.java:542) org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:557) Potentially hanging thread: hconnection-0x34c6ce8e-shared--pool9-t8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: localhost,51420,1428169676736_ChoreService_1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t66 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: M_LOG_REPLAY_OPS-localhost:51327-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t50 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t41 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=439 (was 424) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=219 (was 229), ProcessCount=232 (was 232), AvailableMemoryMB=17 (was 16) - AvailableMemoryMB LEAK? -, ConnectionCount=1 (was 1) 2015-04-04 10:47:59,436 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testHBase3583 Thread=409, OpenFileDescriptor=439, MaxFileDescriptor=10240, SystemLoadAverage=219, ProcessCount=232, AvailableMemoryMB=17, ConnectionCount=1 2015-04-04 10:47:59,439 INFO [B.defaultRpcServer.handler=4,queue=0,port=51327] master.HMaster(1318): Client=apurtell/null create 'testHBase3583', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:47:59,453 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testHBase3583/write-master:513270000000000 2015-04-04 10:47:59,459 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(188): Create table testHBase3583 2015-04-04 10:47:59,469 INFO [IPC Server handler 2 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741857_1033{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 0 2015-04-04 10:47:59,472 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3583/.tabledesc/.tableinfo.0000000001 2015-04-04 10:47:59,473 INFO [RegionOpenAndInitThread-testHBase3583-1] regionserver.HRegion(5885): creating HRegion testHBase3583 HTD == 'testHBase3583', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp Table name == testHBase3583 2015-04-04 10:47:59,482 INFO [IPC Server handler 2 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741858_1034{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 0 2015-04-04 10:47:59,483 DEBUG [RegionOpenAndInitThread-testHBase3583-1] regionserver.HRegion(721): Instantiated testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. 2015-04-04 10:47:59,483 DEBUG [RegionOpenAndInitThread-testHBase3583-1] regionserver.HRegion(1350): Closing testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9.: disabling compactions & flushes 2015-04-04 10:47:59,483 DEBUG [RegionOpenAndInitThread-testHBase3583-1] regionserver.HRegion(1377): Updates disabled for region testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. 2015-04-04 10:47:59,483 INFO [RegionOpenAndInitThread-testHBase3583-1] regionserver.HRegion(1471): Closed testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. 2015-04-04 10:47:59,488 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:47:59,489 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,51330,1428169644428 2015-04-04 10:47:59,489 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKAssign(174): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Async create of unassigned node e34521878297f169be35783d7a7a36b9 with OFFLINE state 2015-04-04 10:47:59,490 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:47:59,491 DEBUG [main-EventThread] master.OfflineCallback(69): rs={e34521878297f169be35783d7a7a36b9 state=OFFLINE, ts=1428169679489, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:47:59,491 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={e34521878297f169be35783d7a7a36b9 state=OFFLINE, ts=1428169679489, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:47:59,496 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1658): localhost,51330,1428169644428 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:47:59,496 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.RegionStates(1112): Transition {e34521878297f169be35783d7a7a36b9 state=OFFLINE, ts=1428169679489, server=null} to {e34521878297f169be35783d7a7a36b9 state=PENDING_OPEN, ts=1428169679496, server=localhost,51330,1428169644428} 2015-04-04 10:47:59,497 INFO [PriorityRpcServer.handler=9,queue=1,port=51330] regionserver.RSRpcServices(1390): Open testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. 2015-04-04 10:47:59,503 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning e34521878297f169be35783d7a7a36b9 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:47:59,504 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1788): Bulk assigning done for localhost,51330,1428169644428 2015-04-04 10:47:59,504 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table testHBase3583 state from ENABLING to ENABLED 2015-04-04 10:47:59,505 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/e34521878297f169be35783d7a7a36b9 2015-04-04 10:47:59,505 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node e34521878297f169be35783d7a7a36b9 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:47:59,505 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(6078): Opening region: {ENCODED => e34521878297f169be35783d7a7a36b9, NAME => 'testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:47:59,506 DEBUG [AM.ZK.Worker-pool3-t10] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,51330,1428169644428, region=e34521878297f169be35783d7a7a36b9, current_state={e34521878297f169be35783d7a7a36b9 state=PENDING_OPEN, ts=1428169679496, server=localhost,51330,1428169644428} 2015-04-04 10:47:59,506 INFO [AM.ZK.Worker-pool3-t10] master.RegionStates(1112): Transition {e34521878297f169be35783d7a7a36b9 state=PENDING_OPEN, ts=1428169679496, server=localhost,51330,1428169644428} to {e34521878297f169be35783d7a7a36b9 state=OPENING, ts=1428169679506, server=localhost,51330,1428169644428} 2015-04-04 10:47:59,506 DEBUG [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:47:59,506 DEBUG [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:47:59,506 INFO [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:47:59,506 INFO [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:47:59,507 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testHBase3583 e34521878297f169be35783d7a7a36b9 2015-04-04 10:47:59,507 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(721): Instantiated testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. 2015-04-04 10:47:59,510 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testHBase3583/write-master:513270000000000 2015-04-04 10:47:59,510 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(218): Table, testHBase3583, creation successful 2015-04-04 10:47:59,510 INFO [StoreOpener-e34521878297f169be35783d7a7a36b9-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:59,511 INFO [StoreOpener-e34521878297f169be35783d7a7a36b9-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:47:59,512 DEBUG [StoreOpener-e34521878297f169be35783d7a7a36b9-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/a 2015-04-04 10:47:59,514 INFO [StoreOpener-e34521878297f169be35783d7a7a36b9-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:59,514 INFO [StoreOpener-e34521878297f169be35783d7a7a36b9-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:47:59,515 DEBUG [StoreOpener-e34521878297f169be35783d7a7a36b9-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/b 2015-04-04 10:47:59,517 INFO [StoreOpener-e34521878297f169be35783d7a7a36b9-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:47:59,517 INFO [StoreOpener-e34521878297f169be35783d7a7a36b9-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:47:59,518 DEBUG [StoreOpener-e34521878297f169be35783d7a7a36b9-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/c 2015-04-04 10:47:59,519 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9 2015-04-04 10:47:59,521 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:47:59,525 DEBUG [RS_OPEN_REGION-localhost:51330-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:47:59,525 INFO [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(847): Onlined e34521878297f169be35783d7a7a36b9; next sequenceid=2 2015-04-04 10:47:59,527 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(643): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Attempting to retransition opening state of node e34521878297f169be35783d7a7a36b9 2015-04-04 10:47:59,528 INFO [PostOpenDeployTasks:e34521878297f169be35783d7a7a36b9] regionserver.HRegionServer(1849): Post open deploy tasks for testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. 2015-04-04 10:47:59,531 INFO [PostOpenDeployTasks:e34521878297f169be35783d7a7a36b9] hbase.MetaTableAccessor(1350): Updated row testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. with server=localhost,51330,1428169644428 2015-04-04 10:47:59,531 DEBUG [PostOpenDeployTasks:e34521878297f169be35783d7a7a36b9] regionserver.HRegionServer(1883): Finished post open deploy task for testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. 2015-04-04 10:47:59,532 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning e34521878297f169be35783d7a7a36b9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:47:59,534 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/e34521878297f169be35783d7a7a36b9 2015-04-04 10:47:59,534 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node e34521878297f169be35783d7a7a36b9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:47:59,534 DEBUG [RS_OPEN_REGION-localhost:51330-0] coordination.ZkOpenRegionCoordination(91): Transitioned e34521878297f169be35783d7a7a36b9 to OPENED in zk on localhost,51330,1428169644428 2015-04-04 10:47:59,534 DEBUG [RS_OPEN_REGION-localhost:51330-0] handler.OpenRegionHandler(167): Opened testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. on localhost,51330,1428169644428 2015-04-04 10:47:59,534 DEBUG [AM.ZK.Worker-pool3-t11] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,51330,1428169644428, region=e34521878297f169be35783d7a7a36b9, current_state={e34521878297f169be35783d7a7a36b9 state=OPENING, ts=1428169679506, server=localhost,51330,1428169644428} 2015-04-04 10:47:59,534 INFO [AM.ZK.Worker-pool3-t11] master.RegionStates(1112): Transition {e34521878297f169be35783d7a7a36b9 state=OPENING, ts=1428169679506, server=localhost,51330,1428169644428} to {e34521878297f169be35783d7a7a36b9 state=OPEN, ts=1428169679534, server=localhost,51330,1428169644428} 2015-04-04 10:47:59,535 DEBUG [AM.ZK.Worker-pool3-t11] coordination.ZkOpenRegionCoordination(350): Handling OPENED of e34521878297f169be35783d7a7a36b9 from localhost,51327,1428169644129; deleting unassigned node 2015-04-04 10:47:59,536 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/e34521878297f169be35783d7a7a36b9 2015-04-04 10:47:59,536 DEBUG [AM.ZK.Worker-pool3-t11] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node e34521878297f169be35783d7a7a36b9 in expected state RS_ZK_REGION_OPENED 2015-04-04 10:47:59,536 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:47:59,536 DEBUG [AM.ZK.Worker-pool3-t11] master.AssignmentManager$4(1337): Znode testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. deleted, state: {e34521878297f169be35783d7a7a36b9 state=OPEN, ts=1428169679534, server=localhost,51330,1428169644428} 2015-04-04 10:47:59,536 DEBUG [AM.ZK.Worker-pool3-t11] master.RegionStates(460): Onlined e34521878297f169be35783d7a7a36b9 on localhost,51330,1428169644428 {ENCODED => e34521878297f169be35783d7a7a36b9, NAME => 'testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:47:59,576 INFO [Thread-410] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:47:59,581 INFO [Thread-410] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:47:59,581 INFO [Thread-410] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:47:59,586 INFO [Thread-410] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:47:59,603 INFO [Thread-410] client.HBaseAdmin$9(979): Started disable of testHBase3583 2015-04-04 10:47:59,603 INFO [B.defaultRpcServer.handler=1,queue=0,port=51327] master.HMaster(1659): Client=apurtell/null disable testHBase3583 2015-04-04 10:47:59,606 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testHBase3583/write-master:513270000000001 2015-04-04 10:47:59,611 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(129): Attempting to disable table testHBase3583 2015-04-04 10:47:59,612 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table testHBase3583 state from DISABLING to DISABLING 2015-04-04 10:47:59,613 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:47:59,614 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. (offlining), current state: {e34521878297f169be35783d7a7a36b9 state=OPEN, ts=1428169679536, server=localhost,51330,1428169644428} 2015-04-04 10:47:59,614 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Creating unassigned node e34521878297f169be35783d7a7a36b9 in a CLOSING state 2015-04-04 10:47:59,614 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:47:59,615 INFO [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {e34521878297f169be35783d7a7a36b9 state=OPEN, ts=1428169679536, server=localhost,51330,1428169644428} to {e34521878297f169be35783d7a7a36b9 state=PENDING_CLOSE, ts=1428169679615, server=localhost,51330,1428169644428} 2015-04-04 10:47:59,615 INFO [PriorityRpcServer.handler=2,queue=0,port=51330] regionserver.RSRpcServices(1018): Close e34521878297f169be35783d7a7a36b9, moving to null 2015-04-04 10:47:59,616 DEBUG [RS_CLOSE_REGION-localhost:51330-2] handler.CloseRegionHandler(119): Processing close of testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. 2015-04-04 10:47:59,616 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,51330,1428169644428 for region testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. 2015-04-04 10:47:59,616 DEBUG [Thread-410] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in testHBase3583 2015-04-04 10:47:59,617 DEBUG [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegion(1350): Closing testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9.: disabling compactions & flushes 2015-04-04 10:47:59,617 DEBUG [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegion(1377): Updates disabled for region testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. 2015-04-04 10:47:59,617 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegion(2084): Started memstore flush for testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9., current region memstore size 160 B, and 3/3 column families' memstores are being flushed. 2015-04-04 10:47:59,627 INFO [IPC Server handler 2 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741859_1035{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 4727 2015-04-04 10:47:59,727 DEBUG [Thread-410] client.HBaseAdmin(1019): Sleeping= 200ms, waiting for all regions to be disabled in testHBase3583 2015-04-04 10:47:59,938 DEBUG [Thread-410] client.HBaseAdmin(1019): Sleeping= 300ms, waiting for all regions to be disabled in testHBase3583 2015-04-04 10:48:00,029 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=6, memsize=160, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/.tmp/35d76bba38c94033841a5f1e3d9cfd8e 2015-04-04 10:48:00,039 DEBUG [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/.tmp/35d76bba38c94033841a5f1e3d9cfd8e as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/a/35d76bba38c94033841a5f1e3d9cfd8e 2015-04-04 10:48:00,047 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/a/35d76bba38c94033841a5f1e3d9cfd8e, entries=1, sequenceid=6, filesize=4.6 K 2015-04-04 10:48:00,048 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegion(2363): Finished memstore flush of ~160 B/160, currentsize=0 B/0 for region testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. in 431ms, sequenceid=6, compaction requested=false 2015-04-04 10:48:00,051 INFO [StoreCloserThread-testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9.-1] regionserver.HStore(879): Closed a 2015-04-04 10:48:00,051 INFO [StoreCloserThread-testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9.-1] regionserver.HStore(879): Closed b 2015-04-04 10:48:00,051 INFO [StoreCloserThread-testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9.-1] regionserver.HStore(879): Closed c 2015-04-04 10:48:00,058 DEBUG [RS_CLOSE_REGION-localhost:51330-2] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/recovered.edits/9.seqid to file, newSeqId=9, maxSeqId=2 2015-04-04 10:48:00,059 DEBUG [RS_CLOSE_REGION-localhost:51330-2] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:48:00,059 DEBUG [RS_CLOSE_REGION-localhost:51330-2] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:48:00,059 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegion(1471): Closed testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. 2015-04-04 10:48:00,060 DEBUG [RS_CLOSE_REGION-localhost:51330-2] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning e34521878297f169be35783d7a7a36b9 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:48:00,061 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/e34521878297f169be35783d7a7a36b9 2015-04-04 10:48:00,061 DEBUG [RS_CLOSE_REGION-localhost:51330-2] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node e34521878297f169be35783d7a7a36b9 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:48:00,061 DEBUG [RS_CLOSE_REGION-localhost:51330-2] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. on localhost,51330,1428169644428 2015-04-04 10:48:00,061 DEBUG [RS_CLOSE_REGION-localhost:51330-2] handler.CloseRegionHandler(164): Closed testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. 2015-04-04 10:48:00,062 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,51330,1428169644428, region=e34521878297f169be35783d7a7a36b9, current_state={e34521878297f169be35783d7a7a36b9 state=PENDING_CLOSE, ts=1428169679615, server=localhost,51330,1428169644428} 2015-04-04 10:48:00,062 DEBUG [AM.ZK.Worker-pool3-t14] handler.ClosedRegionHandler(93): Handling CLOSED event for e34521878297f169be35783d7a7a36b9 2015-04-04 10:48:00,062 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. 2015-04-04 10:48:00,063 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/e34521878297f169be35783d7a7a36b9 2015-04-04 10:48:00,063 DEBUG [AM.ZK.Worker-pool3-t14] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node e34521878297f169be35783d7a7a36b9 in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:48:00,063 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:00,063 INFO [AM.ZK.Worker-pool3-t14] master.RegionStates(1112): Transition {e34521878297f169be35783d7a7a36b9 state=PENDING_CLOSE, ts=1428169679615, server=localhost,51330,1428169644428} to {e34521878297f169be35783d7a7a36b9 state=OFFLINE, ts=1428169680063, server=localhost,51330,1428169644428} 2015-04-04 10:48:00,064 INFO [AM.ZK.Worker-pool3-t14] master.RegionStates(600): Offlined e34521878297f169be35783d7a7a36b9 from localhost,51330,1428169644428 2015-04-04 10:48:00,093 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table testHBase3583 state from DISABLING to DISABLED 2015-04-04 10:48:00,095 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(191): Disabled table, testHBase3583, is done=true 2015-04-04 10:48:00,096 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testHBase3583/write-master:513270000000001 2015-04-04 10:48:00,245 INFO [Thread-410] client.HBaseAdmin(1034): Disabled testHBase3583 2015-04-04 10:48:00,246 INFO [B.defaultRpcServer.handler=4,queue=0,port=51327] master.HMaster(1564): Client=apurtell/null delete testHBase3583 2015-04-04 10:48:00,257 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testHBase3583/write-master:513270000000002 2015-04-04 10:48:00,262 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table testHBase3583 2015-04-04 10:48:00,266 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:48:00,268 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => e34521878297f169be35783d7a7a36b9, NAME => 'testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:48:00,270 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(186): Archiving region testHBase3583,,1428169679439.e34521878297f169be35783d7a7a36b9. from FS 2015-04-04 10:48:00,271 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9 2015-04-04 10:48:00,276 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/recovered.edits] 2015-04-04 10:48:00,286 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/a/35d76bba38c94033841a5f1e3d9cfd8e, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/a/35d76bba38c94033841a5f1e3d9cfd8e 2015-04-04 10:48:00,298 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/recovered.edits/9.seqid, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9/recovered.edits/9.seqid 2015-04-04 10:48:00,299 INFO [IPC Server handler 3 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741858_1034 127.0.0.1:51320 2015-04-04 10:48:00,299 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3583/e34521878297f169be35783d7a7a36b9 2015-04-04 10:48:00,301 INFO [IPC Server handler 0 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741857_1033 127.0.0.1:51320 2015-04-04 10:48:00,301 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(196): Table 'testHBase3583' archived! 2015-04-04 10:48:00,301 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(116): Removing 'testHBase3583' descriptor. 2015-04-04 10:48:00,302 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(122): Removing 'testHBase3583' from region states. 2015-04-04 10:48:00,302 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(126): Marking 'testHBase3583' as deleted. 2015-04-04 10:48:00,309 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testHBase3583/write-master:513270000000002 2015-04-04 10:48:00,375 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51327] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = testHBase3583 org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3583 at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:48:00,375 INFO [Thread-410] client.HBaseAdmin(739): Deleted testHBase3583 2015-04-04 10:48:00,406 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testHBase3583 Thread=433 (was 409) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t84 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t83 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51391 [Waiting for operation #35] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1796537615_1 at /127.0.0.1:51416 [Waiting for operation #11] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x34c6ce8e-shared--pool9-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t88 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t57 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t52 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t82 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t91 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t56 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t89 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51417 [Waiting for operation #10] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t55 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t85 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t58 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t51 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t90 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t80 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t86 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t87 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t53 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t54 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t59 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x130f424b-metaLookup-shared--pool77-t1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t81 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t60 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=439 (was 439), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=219 (was 219), ProcessCount=232 (was 232), AvailableMemoryMB=17 (was 17), ConnectionCount=1 (was 1) 2015-04-04 10:48:00,434 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testHBase3758 Thread=433, OpenFileDescriptor=439, MaxFileDescriptor=10240, SystemLoadAverage=219, ProcessCount=232, AvailableMemoryMB=17, ConnectionCount=1 2015-04-04 10:48:00,437 INFO [B.defaultRpcServer.handler=3,queue=0,port=51327] master.HMaster(1318): Client=apurtell/null create 'testHBase3758', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:48:00,443 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testHBase3758/write-master:513270000000000 2015-04-04 10:48:00,448 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(188): Create table testHBase3758 2015-04-04 10:48:00,458 INFO [IPC Server handler 1 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741860_1036{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 777 2015-04-04 10:48:00,868 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3758/.tabledesc/.tableinfo.0000000001 2015-04-04 10:48:00,870 INFO [RegionOpenAndInitThread-testHBase3758-1] regionserver.HRegion(5885): creating HRegion testHBase3758 HTD == 'testHBase3758', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp Table name == testHBase3758 2015-04-04 10:48:00,881 INFO [IPC Server handler 1 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741861_1037{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 0 2015-04-04 10:48:00,883 DEBUG [RegionOpenAndInitThread-testHBase3758-1] regionserver.HRegion(721): Instantiated testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. 2015-04-04 10:48:00,883 DEBUG [RegionOpenAndInitThread-testHBase3758-1] regionserver.HRegion(1350): Closing testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f.: disabling compactions & flushes 2015-04-04 10:48:00,883 DEBUG [RegionOpenAndInitThread-testHBase3758-1] regionserver.HRegion(1377): Updates disabled for region testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. 2015-04-04 10:48:00,883 INFO [RegionOpenAndInitThread-testHBase3758-1] regionserver.HRegion(1471): Closed testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. 2015-04-04 10:48:00,887 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:48:00,888 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,51330,1428169644428 2015-04-04 10:48:00,888 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKAssign(174): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Async create of unassigned node c77b598b15ee1268f92110fcd670332f with OFFLINE state 2015-04-04 10:48:00,889 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:00,889 DEBUG [main-EventThread] master.OfflineCallback(69): rs={c77b598b15ee1268f92110fcd670332f state=OFFLINE, ts=1428169680888, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:48:00,890 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={c77b598b15ee1268f92110fcd670332f state=OFFLINE, ts=1428169680888, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:48:00,894 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1658): localhost,51330,1428169644428 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:48:00,894 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.RegionStates(1112): Transition {c77b598b15ee1268f92110fcd670332f state=OFFLINE, ts=1428169680888, server=null} to {c77b598b15ee1268f92110fcd670332f state=PENDING_OPEN, ts=1428169680894, server=localhost,51330,1428169644428} 2015-04-04 10:48:00,895 INFO [PriorityRpcServer.handler=2,queue=0,port=51330] regionserver.RSRpcServices(1390): Open testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. 2015-04-04 10:48:00,901 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning c77b598b15ee1268f92110fcd670332f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:48:00,901 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1788): Bulk assigning done for localhost,51330,1428169644428 2015-04-04 10:48:00,901 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table testHBase3758 state from ENABLING to ENABLED 2015-04-04 10:48:00,903 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/c77b598b15ee1268f92110fcd670332f 2015-04-04 10:48:00,903 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node c77b598b15ee1268f92110fcd670332f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:48:00,904 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(6078): Opening region: {ENCODED => c77b598b15ee1268f92110fcd670332f, NAME => 'testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:48:00,904 DEBUG [AM.ZK.Worker-pool3-t18] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,51330,1428169644428, region=c77b598b15ee1268f92110fcd670332f, current_state={c77b598b15ee1268f92110fcd670332f state=PENDING_OPEN, ts=1428169680894, server=localhost,51330,1428169644428} 2015-04-04 10:48:00,904 INFO [AM.ZK.Worker-pool3-t18] master.RegionStates(1112): Transition {c77b598b15ee1268f92110fcd670332f state=PENDING_OPEN, ts=1428169680894, server=localhost,51330,1428169644428} to {c77b598b15ee1268f92110fcd670332f state=OPENING, ts=1428169680904, server=localhost,51330,1428169644428} 2015-04-04 10:48:00,904 DEBUG [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:48:00,904 DEBUG [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:48:00,904 INFO [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:48:00,905 INFO [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:48:00,905 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table testHBase3758 c77b598b15ee1268f92110fcd670332f 2015-04-04 10:48:00,905 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(721): Instantiated testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. 2015-04-04 10:48:00,909 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testHBase3758/write-master:513270000000000 2015-04-04 10:48:00,909 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(218): Table, testHBase3758, creation successful 2015-04-04 10:48:00,909 INFO [StoreOpener-c77b598b15ee1268f92110fcd670332f-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:00,910 INFO [StoreOpener-c77b598b15ee1268f92110fcd670332f-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:00,911 DEBUG [StoreOpener-c77b598b15ee1268f92110fcd670332f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/a 2015-04-04 10:48:00,913 INFO [StoreOpener-c77b598b15ee1268f92110fcd670332f-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:00,913 INFO [StoreOpener-c77b598b15ee1268f92110fcd670332f-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:00,914 DEBUG [StoreOpener-c77b598b15ee1268f92110fcd670332f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/b 2015-04-04 10:48:00,916 INFO [StoreOpener-c77b598b15ee1268f92110fcd670332f-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:00,917 INFO [StoreOpener-c77b598b15ee1268f92110fcd670332f-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:00,918 DEBUG [StoreOpener-c77b598b15ee1268f92110fcd670332f-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/c 2015-04-04 10:48:00,919 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f 2015-04-04 10:48:00,922 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:48:00,928 DEBUG [RS_OPEN_REGION-localhost:51330-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:48:00,928 INFO [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(847): Onlined c77b598b15ee1268f92110fcd670332f; next sequenceid=2 2015-04-04 10:48:00,929 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(643): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Attempting to retransition opening state of node c77b598b15ee1268f92110fcd670332f 2015-04-04 10:48:00,931 INFO [PostOpenDeployTasks:c77b598b15ee1268f92110fcd670332f] regionserver.HRegionServer(1849): Post open deploy tasks for testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. 2015-04-04 10:48:00,934 INFO [PostOpenDeployTasks:c77b598b15ee1268f92110fcd670332f] hbase.MetaTableAccessor(1350): Updated row testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. with server=localhost,51330,1428169644428 2015-04-04 10:48:00,934 DEBUG [PostOpenDeployTasks:c77b598b15ee1268f92110fcd670332f] regionserver.HRegionServer(1883): Finished post open deploy task for testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. 2015-04-04 10:48:00,935 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning c77b598b15ee1268f92110fcd670332f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:48:00,936 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/c77b598b15ee1268f92110fcd670332f 2015-04-04 10:48:00,936 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node c77b598b15ee1268f92110fcd670332f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:48:00,936 DEBUG [RS_OPEN_REGION-localhost:51330-1] coordination.ZkOpenRegionCoordination(91): Transitioned c77b598b15ee1268f92110fcd670332f to OPENED in zk on localhost,51330,1428169644428 2015-04-04 10:48:00,936 DEBUG [RS_OPEN_REGION-localhost:51330-1] handler.OpenRegionHandler(167): Opened testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. on localhost,51330,1428169644428 2015-04-04 10:48:00,937 DEBUG [AM.ZK.Worker-pool3-t3] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,51330,1428169644428, region=c77b598b15ee1268f92110fcd670332f, current_state={c77b598b15ee1268f92110fcd670332f state=OPENING, ts=1428169680904, server=localhost,51330,1428169644428} 2015-04-04 10:48:00,937 INFO [AM.ZK.Worker-pool3-t3] master.RegionStates(1112): Transition {c77b598b15ee1268f92110fcd670332f state=OPENING, ts=1428169680904, server=localhost,51330,1428169644428} to {c77b598b15ee1268f92110fcd670332f state=OPEN, ts=1428169680937, server=localhost,51330,1428169644428} 2015-04-04 10:48:00,937 DEBUG [AM.ZK.Worker-pool3-t3] coordination.ZkOpenRegionCoordination(350): Handling OPENED of c77b598b15ee1268f92110fcd670332f from localhost,51327,1428169644129; deleting unassigned node 2015-04-04 10:48:00,938 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/c77b598b15ee1268f92110fcd670332f 2015-04-04 10:48:00,938 DEBUG [AM.ZK.Worker-pool3-t3] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node c77b598b15ee1268f92110fcd670332f in expected state RS_ZK_REGION_OPENED 2015-04-04 10:48:00,938 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:00,938 DEBUG [AM.ZK.Worker-pool3-t3] master.AssignmentManager$4(1337): Znode testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. deleted, state: {c77b598b15ee1268f92110fcd670332f state=OPEN, ts=1428169680937, server=localhost,51330,1428169644428} 2015-04-04 10:48:00,938 DEBUG [AM.ZK.Worker-pool3-t3] master.RegionStates(460): Onlined c77b598b15ee1268f92110fcd670332f on localhost,51330,1428169644428 {ENCODED => c77b598b15ee1268f92110fcd670332f, NAME => 'testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:48:01,079 INFO [Thread-425] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:48:01,084 INFO [Thread-425] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:48:01,096 INFO [Thread-425] client.HBaseAdmin$9(979): Started disable of testHBase3758 2015-04-04 10:48:01,097 INFO [B.defaultRpcServer.handler=1,queue=0,port=51327] master.HMaster(1659): Client=apurtell/null disable testHBase3758 2015-04-04 10:48:01,107 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testHBase3758/write-master:513270000000001 2015-04-04 10:48:01,114 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(129): Attempting to disable table testHBase3758 2015-04-04 10:48:01,115 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table testHBase3758 state from DISABLING to DISABLING 2015-04-04 10:48:01,116 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:48:01,117 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. (offlining), current state: {c77b598b15ee1268f92110fcd670332f state=OPEN, ts=1428169680938, server=localhost,51330,1428169644428} 2015-04-04 10:48:01,117 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Creating unassigned node c77b598b15ee1268f92110fcd670332f in a CLOSING state 2015-04-04 10:48:01,118 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:01,118 INFO [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {c77b598b15ee1268f92110fcd670332f state=OPEN, ts=1428169680938, server=localhost,51330,1428169644428} to {c77b598b15ee1268f92110fcd670332f state=PENDING_CLOSE, ts=1428169681118, server=localhost,51330,1428169644428} 2015-04-04 10:48:01,119 INFO [PriorityRpcServer.handler=9,queue=1,port=51330] regionserver.RSRpcServices(1018): Close c77b598b15ee1268f92110fcd670332f, moving to null 2015-04-04 10:48:01,119 DEBUG [RS_CLOSE_REGION-localhost:51330-0] handler.CloseRegionHandler(119): Processing close of testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. 2015-04-04 10:48:01,119 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,51330,1428169644428 for region testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. 2015-04-04 10:48:01,120 DEBUG [Thread-425] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in testHBase3758 2015-04-04 10:48:01,120 DEBUG [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(1350): Closing testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f.: disabling compactions & flushes 2015-04-04 10:48:01,120 DEBUG [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(1377): Updates disabled for region testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. 2015-04-04 10:48:01,120 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(2084): Started memstore flush for testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f., current region memstore size 640 B, and 3/3 column families' memstores are being flushed. 2015-04-04 10:48:01,132 INFO [IPC Server handler 1 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741862_1038{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 0 2015-04-04 10:48:01,132 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/.tmp/9fa35ea9a5084581853ebb5e8c709ad7 2015-04-04 10:48:01,139 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9fa35ea9a5084581853ebb5e8c709ad7 2015-04-04 10:48:01,148 INFO [IPC Server handler 0 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741863_1039{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 4911 2015-04-04 10:48:01,230 DEBUG [Thread-425] client.HBaseAdmin(1019): Sleeping= 200ms, waiting for all regions to be disabled in testHBase3758 2015-04-04 10:48:01,436 DEBUG [Thread-425] client.HBaseAdmin(1019): Sleeping= 300ms, waiting for all regions to be disabled in testHBase3758 2015-04-04 10:48:01,551 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=160, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/.tmp/fc9043b03bf542539b3a3c3f8279062c 2015-04-04 10:48:01,560 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fc9043b03bf542539b3a3c3f8279062c 2015-04-04 10:48:01,573 INFO [IPC Server handler 2 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741864_1040{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 0 2015-04-04 10:48:01,573 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=160, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/.tmp/5a806019c8a64a5da55e58b633750a66 2015-04-04 10:48:01,582 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5a806019c8a64a5da55e58b633750a66 2015-04-04 10:48:01,584 DEBUG [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/.tmp/9fa35ea9a5084581853ebb5e8c709ad7 as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/a/9fa35ea9a5084581853ebb5e8c709ad7 2015-04-04 10:48:01,592 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9fa35ea9a5084581853ebb5e8c709ad7 2015-04-04 10:48:01,593 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/a/9fa35ea9a5084581853ebb5e8c709ad7, entries=1, sequenceid=8, filesize=4.8 K 2015-04-04 10:48:01,594 DEBUG [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/.tmp/fc9043b03bf542539b3a3c3f8279062c as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/b/fc9043b03bf542539b3a3c3f8279062c 2015-04-04 10:48:01,603 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fc9043b03bf542539b3a3c3f8279062c 2015-04-04 10:48:01,603 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/b/fc9043b03bf542539b3a3c3f8279062c, entries=1, sequenceid=8, filesize=4.8 K 2015-04-04 10:48:01,605 DEBUG [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/.tmp/5a806019c8a64a5da55e58b633750a66 as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/c/5a806019c8a64a5da55e58b633750a66 2015-04-04 10:48:01,613 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5a806019c8a64a5da55e58b633750a66 2015-04-04 10:48:01,613 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/c/5a806019c8a64a5da55e58b633750a66, entries=1, sequenceid=8, filesize=4.8 K 2015-04-04 10:48:01,614 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(2363): Finished memstore flush of ~640 B/640, currentsize=0 B/0 for region testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. in 494ms, sequenceid=8, compaction requested=false 2015-04-04 10:48:01,616 INFO [StoreCloserThread-testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f.-1] regionserver.HStore(879): Closed a 2015-04-04 10:48:01,618 INFO [StoreCloserThread-testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f.-1] regionserver.HStore(879): Closed b 2015-04-04 10:48:01,619 INFO [StoreCloserThread-testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f.-1] regionserver.HStore(879): Closed c 2015-04-04 10:48:01,628 DEBUG [RS_CLOSE_REGION-localhost:51330-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/recovered.edits/11.seqid to file, newSeqId=11, maxSeqId=2 2015-04-04 10:48:01,629 DEBUG [RS_CLOSE_REGION-localhost:51330-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:48:01,629 DEBUG [RS_CLOSE_REGION-localhost:51330-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:48:01,629 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(1471): Closed testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. 2015-04-04 10:48:01,629 DEBUG [RS_CLOSE_REGION-localhost:51330-0] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning c77b598b15ee1268f92110fcd670332f from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:48:01,631 DEBUG [RS_CLOSE_REGION-localhost:51330-0] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node c77b598b15ee1268f92110fcd670332f from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:48:01,631 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/c77b598b15ee1268f92110fcd670332f 2015-04-04 10:48:01,631 DEBUG [RS_CLOSE_REGION-localhost:51330-0] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. on localhost,51330,1428169644428 2015-04-04 10:48:01,631 DEBUG [RS_CLOSE_REGION-localhost:51330-0] handler.CloseRegionHandler(164): Closed testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. 2015-04-04 10:48:01,631 DEBUG [AM.ZK.Worker-pool3-t9] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,51330,1428169644428, region=c77b598b15ee1268f92110fcd670332f, current_state={c77b598b15ee1268f92110fcd670332f state=PENDING_CLOSE, ts=1428169681118, server=localhost,51330,1428169644428} 2015-04-04 10:48:01,631 DEBUG [AM.ZK.Worker-pool3-t9] handler.ClosedRegionHandler(93): Handling CLOSED event for c77b598b15ee1268f92110fcd670332f 2015-04-04 10:48:01,631 DEBUG [AM.ZK.Worker-pool3-t9] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. 2015-04-04 10:48:01,633 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/c77b598b15ee1268f92110fcd670332f 2015-04-04 10:48:01,633 DEBUG [AM.ZK.Worker-pool3-t9] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node c77b598b15ee1268f92110fcd670332f in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:48:01,633 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:01,633 INFO [AM.ZK.Worker-pool3-t9] master.RegionStates(1112): Transition {c77b598b15ee1268f92110fcd670332f state=PENDING_CLOSE, ts=1428169681118, server=localhost,51330,1428169644428} to {c77b598b15ee1268f92110fcd670332f state=OFFLINE, ts=1428169681633, server=localhost,51330,1428169644428} 2015-04-04 10:48:01,633 INFO [AM.ZK.Worker-pool3-t9] master.RegionStates(600): Offlined c77b598b15ee1268f92110fcd670332f from localhost,51330,1428169644428 2015-04-04 10:48:01,653 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table testHBase3758 state from DISABLING to DISABLED 2015-04-04 10:48:01,655 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(191): Disabled table, testHBase3758, is done=true 2015-04-04 10:48:01,656 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testHBase3758/write-master:513270000000001 2015-04-04 10:48:01,745 INFO [Thread-425] client.HBaseAdmin(1034): Disabled testHBase3758 2015-04-04 10:48:01,746 INFO [B.defaultRpcServer.handler=3,queue=0,port=51327] master.HMaster(1564): Client=apurtell/null delete testHBase3758 2015-04-04 10:48:01,747 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testHBase3758/write-master:513270000000002 2015-04-04 10:48:01,752 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table testHBase3758 2015-04-04 10:48:01,755 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:48:01,758 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => c77b598b15ee1268f92110fcd670332f, NAME => 'testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:48:01,760 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(186): Archiving region testHBase3758,,1428169680436.c77b598b15ee1268f92110fcd670332f. from FS 2015-04-04 10:48:01,760 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f 2015-04-04 10:48:01,765 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/recovered.edits] 2015-04-04 10:48:01,773 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/a/9fa35ea9a5084581853ebb5e8c709ad7, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/a/9fa35ea9a5084581853ebb5e8c709ad7 2015-04-04 10:48:01,779 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/b/fc9043b03bf542539b3a3c3f8279062c, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/b/fc9043b03bf542539b3a3c3f8279062c 2015-04-04 10:48:01,785 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/c/5a806019c8a64a5da55e58b633750a66, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/c/5a806019c8a64a5da55e58b633750a66 2015-04-04 10:48:01,791 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/recovered.edits/11.seqid, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f/recovered.edits/11.seqid 2015-04-04 10:48:01,792 INFO [IPC Server handler 6 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741861_1037 127.0.0.1:51320 2015-04-04 10:48:01,792 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/testHBase3758/c77b598b15ee1268f92110fcd670332f 2015-04-04 10:48:01,793 INFO [IPC Server handler 3 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741860_1036 127.0.0.1:51320 2015-04-04 10:48:01,793 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(196): Table 'testHBase3758' archived! 2015-04-04 10:48:01,793 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(116): Removing 'testHBase3758' descriptor. 2015-04-04 10:48:01,794 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(122): Removing 'testHBase3758' from region states. 2015-04-04 10:48:01,794 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(126): Marking 'testHBase3758' as deleted. 2015-04-04 10:48:01,800 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testHBase3758/write-master:513270000000002 2015-04-04 10:48:01,867 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51327] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = testHBase3758 org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/testHBase3758 at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:48:01,867 INFO [Thread-425] client.HBaseAdmin(739): Deleted testHBase3758 2015-04-04 10:48:01,900 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testHBase3758 Thread=457 (was 433) Potentially hanging thread: hconnection-0x34c6ce8e-shared--pool9-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t65 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t62 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51391 [Waiting for operation #39] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51416 [Waiting for operation #17] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t70 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t64 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t98 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t92 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t97 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t69 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t101 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t93 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t66 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t63 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t96 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t102 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51417 [Waiting for operation #14] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t100 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t95 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x130f424b-metaLookup-shared--pool77-t2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t67 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t68 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t94 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t99 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t61 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t103 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=439 (was 439), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=217 (was 219), ProcessCount=232 (was 232), AvailableMemoryMB=18 (was 17) - AvailableMemoryMB LEAK? -, ConnectionCount=1 (was 1) 2015-04-04 10:48:01,932 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testCompactionOverride Thread=457, OpenFileDescriptor=439, MaxFileDescriptor=10240, SystemLoadAverage=217, ProcessCount=232, AvailableMemoryMB=18, ConnectionCount=1 2015-04-04 10:48:01,939 INFO [B.defaultRpcServer.handler=3,queue=0,port=51327] master.HMaster(1318): Client=apurtell/null create 'TestCompactionOverride', {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor|1073741823|'}, {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:48:01,943 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestCompactionOverride/write-master:513270000000000 2015-04-04 10:48:01,950 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(188): Create table TestCompactionOverride 2015-04-04 10:48:01,959 INFO [IPC Server handler 7 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741865_1041{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 412 2015-04-04 10:48:02,366 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestCompactionOverride/.tabledesc/.tableinfo.0000000001 2015-04-04 10:48:02,367 INFO [RegionOpenAndInitThread-TestCompactionOverride-1] regionserver.HRegion(5885): creating HRegion TestCompactionOverride HTD == 'TestCompactionOverride', {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor|1073741823|'}, {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp Table name == TestCompactionOverride 2015-04-04 10:48:02,376 INFO [IPC Server handler 3 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741866_1042{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 0 2015-04-04 10:48:02,377 DEBUG [RegionOpenAndInitThread-TestCompactionOverride-1] regionserver.HRegion(721): Instantiated TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. 2015-04-04 10:48:02,377 DEBUG [RegionOpenAndInitThread-TestCompactionOverride-1] regionserver.HRegion(1350): Closing TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46.: disabling compactions & flushes 2015-04-04 10:48:02,377 DEBUG [RegionOpenAndInitThread-TestCompactionOverride-1] regionserver.HRegion(1377): Updates disabled for region TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. 2015-04-04 10:48:02,377 INFO [RegionOpenAndInitThread-TestCompactionOverride-1] regionserver.HRegion(1471): Closed TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. 2015-04-04 10:48:02,381 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:48:02,382 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,51330,1428169644428 2015-04-04 10:48:02,382 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKAssign(174): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Async create of unassigned node 1d8e5c1a351aa112c824cf548d63bb46 with OFFLINE state 2015-04-04 10:48:02,383 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:02,383 DEBUG [main-EventThread] master.OfflineCallback(69): rs={1d8e5c1a351aa112c824cf548d63bb46 state=OFFLINE, ts=1428169682382, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:48:02,384 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={1d8e5c1a351aa112c824cf548d63bb46 state=OFFLINE, ts=1428169682382, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:48:02,389 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1658): localhost,51330,1428169644428 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:48:02,389 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.RegionStates(1112): Transition {1d8e5c1a351aa112c824cf548d63bb46 state=OFFLINE, ts=1428169682382, server=null} to {1d8e5c1a351aa112c824cf548d63bb46 state=PENDING_OPEN, ts=1428169682389, server=localhost,51330,1428169644428} 2015-04-04 10:48:02,390 INFO [PriorityRpcServer.handler=8,queue=0,port=51330] regionserver.RSRpcServices(1390): Open TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. 2015-04-04 10:48:02,396 DEBUG [RS_OPEN_REGION-localhost:51330-2] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning 1d8e5c1a351aa112c824cf548d63bb46 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:48:02,396 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1788): Bulk assigning done for localhost,51330,1428169644428 2015-04-04 10:48:02,396 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestCompactionOverride state from ENABLING to ENABLED 2015-04-04 10:48:02,398 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1d8e5c1a351aa112c824cf548d63bb46 2015-04-04 10:48:02,398 DEBUG [RS_OPEN_REGION-localhost:51330-2] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node 1d8e5c1a351aa112c824cf548d63bb46 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:48:02,398 DEBUG [RS_OPEN_REGION-localhost:51330-2] regionserver.HRegion(6078): Opening region: {ENCODED => 1d8e5c1a351aa112c824cf548d63bb46, NAME => 'TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:48:02,398 DEBUG [AM.ZK.Worker-pool3-t7] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,51330,1428169644428, region=1d8e5c1a351aa112c824cf548d63bb46, current_state={1d8e5c1a351aa112c824cf548d63bb46 state=PENDING_OPEN, ts=1428169682389, server=localhost,51330,1428169644428} 2015-04-04 10:48:02,399 INFO [AM.ZK.Worker-pool3-t7] master.RegionStates(1112): Transition {1d8e5c1a351aa112c824cf548d63bb46 state=PENDING_OPEN, ts=1428169682389, server=localhost,51330,1428169644428} to {1d8e5c1a351aa112c824cf548d63bb46 state=OPENING, ts=1428169682399, server=localhost,51330,1428169644428} 2015-04-04 10:48:02,399 DEBUG [RS_OPEN_REGION-localhost:51330-2] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:48:02,399 DEBUG [RS_OPEN_REGION-localhost:51330-2] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:48:02,399 INFO [RS_OPEN_REGION-localhost:51330-2] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:48:02,399 INFO [RS_OPEN_REGION-localhost:51330-2] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:48:02,401 DEBUG [RS_OPEN_REGION-localhost:51330-2] coprocessor.CoprocessorHost(180): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor with path null and priority 1073741823 2015-04-04 10:48:02,401 INFO [RS_OPEN_REGION-localhost:51330-2] regionserver.RegionCoprocessorHost(350): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor from HTD of TestCompactionOverride successfully. 2015-04-04 10:48:02,401 DEBUG [RS_OPEN_REGION-localhost:51330-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestCompactionOverride 1d8e5c1a351aa112c824cf548d63bb46 2015-04-04 10:48:02,402 DEBUG [RS_OPEN_REGION-localhost:51330-2] regionserver.HRegion(721): Instantiated TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. 2015-04-04 10:48:02,402 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestCompactionOverride/write-master:513270000000000 2015-04-04 10:48:02,403 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(218): Table, TestCompactionOverride, creation successful 2015-04-04 10:48:02,407 INFO [StoreOpener-1d8e5c1a351aa112c824cf548d63bb46-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:02,407 INFO [StoreOpener-1d8e5c1a351aa112c824cf548d63bb46-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:02,408 DEBUG [StoreOpener-1d8e5c1a351aa112c824cf548d63bb46-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestCompactionOverride/1d8e5c1a351aa112c824cf548d63bb46/a 2015-04-04 10:48:02,409 DEBUG [RS_OPEN_REGION-localhost:51330-2] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestCompactionOverride/1d8e5c1a351aa112c824cf548d63bb46 2015-04-04 10:48:02,411 DEBUG [RS_OPEN_REGION-localhost:51330-2] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:48:02,415 DEBUG [RS_OPEN_REGION-localhost:51330-2] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestCompactionOverride/1d8e5c1a351aa112c824cf548d63bb46/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:48:02,415 INFO [RS_OPEN_REGION-localhost:51330-2] regionserver.HRegion(847): Onlined 1d8e5c1a351aa112c824cf548d63bb46; next sequenceid=2 2015-04-04 10:48:02,417 DEBUG [RS_OPEN_REGION-localhost:51330-2] zookeeper.ZKAssign(643): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Attempting to retransition opening state of node 1d8e5c1a351aa112c824cf548d63bb46 2015-04-04 10:48:02,417 INFO [PostOpenDeployTasks:1d8e5c1a351aa112c824cf548d63bb46] regionserver.HRegionServer(1849): Post open deploy tasks for TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. 2015-04-04 10:48:02,420 INFO [PostOpenDeployTasks:1d8e5c1a351aa112c824cf548d63bb46] hbase.MetaTableAccessor(1350): Updated row TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. with server=localhost,51330,1428169644428 2015-04-04 10:48:02,421 DEBUG [PostOpenDeployTasks:1d8e5c1a351aa112c824cf548d63bb46] regionserver.HRegionServer(1883): Finished post open deploy task for TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. 2015-04-04 10:48:02,421 DEBUG [RS_OPEN_REGION-localhost:51330-2] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning 1d8e5c1a351aa112c824cf548d63bb46 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:48:02,423 DEBUG [RS_OPEN_REGION-localhost:51330-2] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node 1d8e5c1a351aa112c824cf548d63bb46 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:48:02,423 DEBUG [RS_OPEN_REGION-localhost:51330-2] coordination.ZkOpenRegionCoordination(91): Transitioned 1d8e5c1a351aa112c824cf548d63bb46 to OPENED in zk on localhost,51330,1428169644428 2015-04-04 10:48:02,423 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1d8e5c1a351aa112c824cf548d63bb46 2015-04-04 10:48:02,423 DEBUG [RS_OPEN_REGION-localhost:51330-2] handler.OpenRegionHandler(167): Opened TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. on localhost,51330,1428169644428 2015-04-04 10:48:02,424 DEBUG [AM.ZK.Worker-pool3-t12] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,51330,1428169644428, region=1d8e5c1a351aa112c824cf548d63bb46, current_state={1d8e5c1a351aa112c824cf548d63bb46 state=OPENING, ts=1428169682399, server=localhost,51330,1428169644428} 2015-04-04 10:48:02,424 INFO [AM.ZK.Worker-pool3-t12] master.RegionStates(1112): Transition {1d8e5c1a351aa112c824cf548d63bb46 state=OPENING, ts=1428169682399, server=localhost,51330,1428169644428} to {1d8e5c1a351aa112c824cf548d63bb46 state=OPEN, ts=1428169682424, server=localhost,51330,1428169644428} 2015-04-04 10:48:02,424 DEBUG [AM.ZK.Worker-pool3-t12] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 1d8e5c1a351aa112c824cf548d63bb46 from localhost,51327,1428169644129; deleting unassigned node 2015-04-04 10:48:02,425 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/1d8e5c1a351aa112c824cf548d63bb46 2015-04-04 10:48:02,425 DEBUG [AM.ZK.Worker-pool3-t12] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node 1d8e5c1a351aa112c824cf548d63bb46 in expected state RS_ZK_REGION_OPENED 2015-04-04 10:48:02,425 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:02,426 DEBUG [AM.ZK.Worker-pool3-t12] master.AssignmentManager$4(1337): Znode TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. deleted, state: {1d8e5c1a351aa112c824cf548d63bb46 state=OPEN, ts=1428169682424, server=localhost,51330,1428169644428} 2015-04-04 10:48:02,426 DEBUG [AM.ZK.Worker-pool3-t12] master.RegionStates(460): Onlined 1d8e5c1a351aa112c824cf548d63bb46 on localhost,51330,1428169644428 {ENCODED => 1d8e5c1a351aa112c824cf548d63bb46, NAME => 'TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:48:02,597 INFO [B.defaultRpcServer.handler=2,queue=0,port=51330] regionserver.HRegion(7587): writing data to region TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. with WAL disabled. Data may be lost in the event of a crash. 2015-04-04 10:48:02,632 INFO [B.defaultRpcServer.handler=0,queue=0,port=51327] master.MasterRpcServices(648): Client=apurtell/null procedure request for: flush-table-proc 2015-04-04 10:48:02,641 INFO [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] procedure.Procedure(196): Starting procedure 'TestCompactionOverride' 2015-04-04 10:48:02,641 DEBUG [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2015-04-04 10:48:02,642 DEBUG [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] procedure.Procedure(204): Procedure 'TestCompactionOverride' starting 'acquire' 2015-04-04 10:48:02,642 DEBUG [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] procedure.Procedure(246): Starting procedure 'TestCompactionOverride', kicking off acquire phase on members. 2015-04-04 10:48:02,643 DEBUG [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] zookeeper.ZKUtil(492): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestCompactionOverride 2015-04-04 10:48:02,643 DEBUG [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] procedure.ZKProcedureCoordinatorRpcs(93): Creating acquire znode:/hbase/flush-table-proc/acquired/TestCompactionOverride 2015-04-04 10:48:02,644 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2015-04-04 10:48:02,644 DEBUG [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] procedure.ZKProcedureCoordinatorRpcs(101): Watching for acquire node:/hbase/flush-table-proc/acquired/TestCompactionOverride/localhost,51330,1428169644428 2015-04-04 10:48:02,644 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2015-04-04 10:48:02,644 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-04-04 10:48:02,645 DEBUG [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] zookeeper.ZKUtil(492): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestCompactionOverride/localhost,51330,1428169644428 2015-04-04 10:48:02,645 DEBUG [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2015-04-04 10:48:02,645 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/TestCompactionOverride 2015-04-04 10:48:02,645 DEBUG [main-EventThread] zookeeper.ZKUtil(492): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestCompactionOverride 2015-04-04 10:48:02,646 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2015-04-04 10:48:02,646 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/TestCompactionOverride 2015-04-04 10:48:02,646 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(151): Launching subprocedure to flush regions for TestCompactionOverride 2015-04-04 10:48:02,649 DEBUG [main-EventThread] procedure.ProcedureMember(154): Submitting new Subprocedure:TestCompactionOverride 2015-04-04 10:48:02,649 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] procedure.Subprocedure(152): Starting subprocedure 'TestCompactionOverride' with timeout 60000ms 2015-04-04 10:48:02,649 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2015-04-04 10:48:02,650 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] procedure.Subprocedure(160): Subprocedure 'TestCompactionOverride' starting 'acquire' stage 2015-04-04 10:48:02,651 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2015-04-04 10:48:02,651 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(239): Waiting for local region flush to finish. 2015-04-04 10:48:02,651 DEBUG [rs(localhost,51330,1428169644428)-flush-proc-pool11-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(65): Starting region operation on TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. 2015-04-04 10:48:02,652 DEBUG [rs(localhost,51330,1428169644428)-flush-proc-pool11-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(68): Flush region TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. started... 2015-04-04 10:48:02,652 INFO [rs(localhost,51330,1428169644428)-flush-proc-pool11-thread-1] regionserver.HRegion(2084): Started memstore flush for TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46., current region memstore size 1.64 KB, and 1/1 column families' memstores are being flushed. 2015-04-04 10:48:02,652 WARN [rs(localhost,51330,1428169644428)-flush-proc-pool11-thread-1] wal.FSHLog(1669): Couldn't find oldest seqNum for the region we are about to flush: [1d8e5c1a351aa112c824cf548d63bb46] 2015-04-04 10:48:02,665 INFO [IPC Server handler 2 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741867_1043{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 0 2015-04-04 10:48:02,666 INFO [rs(localhost,51330,1428169644428)-flush-proc-pool11-thread-1] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=24, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestCompactionOverride/1d8e5c1a351aa112c824cf548d63bb46/.tmp/c8545aca9121486d9414ba50a6613cd4 2015-04-04 10:48:02,674 DEBUG [rs(localhost,51330,1428169644428)-flush-proc-pool11-thread-1] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestCompactionOverride/1d8e5c1a351aa112c824cf548d63bb46/.tmp/c8545aca9121486d9414ba50a6613cd4 as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestCompactionOverride/1d8e5c1a351aa112c824cf548d63bb46/a/c8545aca9121486d9414ba50a6613cd4 2015-04-04 10:48:02,682 INFO [rs(localhost,51330,1428169644428)-flush-proc-pool11-thread-1] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestCompactionOverride/1d8e5c1a351aa112c824cf548d63bb46/a/c8545aca9121486d9414ba50a6613cd4, entries=10, sequenceid=24, filesize=5.2 K 2015-04-04 10:48:02,683 INFO [rs(localhost,51330,1428169644428)-flush-proc-pool11-thread-1] regionserver.HRegion(2363): Finished memstore flush of ~1.64 KB/1680, currentsize=0 B/0 for region TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. in 31ms, sequenceid=24, compaction requested=false 2015-04-04 10:48:02,683 DEBUG [rs(localhost,51330,1428169644428)-flush-proc-pool11-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. 2015-04-04 10:48:02,684 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(250): Completed 1/1 local region flush tasks. 2015-04-04 10:48:02,684 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(252): Completed 1 local region flush tasks. 2015-04-04 10:48:02,684 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(280): cancelling 0 flush region tasks localhost,51330,1428169644428 2015-04-04 10:48:02,684 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] procedure.Subprocedure(162): Subprocedure 'TestCompactionOverride' locally acquired 2015-04-04 10:48:02,684 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'localhost,51330,1428169644428' joining acquired barrier for procedure (TestCompactionOverride) in zk 2015-04-04 10:48:02,686 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestCompactionOverride/localhost,51330,1428169644428 2015-04-04 10:48:02,686 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestCompactionOverride 2015-04-04 10:48:02,686 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(186): Node created: /hbase/flush-table-proc/acquired/TestCompactionOverride/localhost,51330,1428169644428 2015-04-04 10:48:02,686 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2015-04-04 10:48:02,686 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2015-04-04 10:48:02,686 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] zookeeper.ZKUtil(492): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestCompactionOverride 2015-04-04 10:48:02,686 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] procedure.Subprocedure(167): Subprocedure 'TestCompactionOverride' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2015-04-04 10:48:02,687 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2015-04-04 10:48:02,687 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestCompactionOverride 2015-04-04 10:48:02,687 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------localhost,51330,1428169644428 2015-04-04 10:48:02,688 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2015-04-04 10:48:02,688 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2015-04-04 10:48:02,689 DEBUG [main-EventThread] procedure.Procedure(297): member: 'localhost,51330,1428169644428' joining acquired barrier for procedure 'TestCompactionOverride' on coordinator 2015-04-04 10:48:02,689 DEBUG [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] procedure.Procedure(212): Procedure 'TestCompactionOverride' starting 'in-barrier' execution. 2015-04-04 10:48:02,689 DEBUG [main-EventThread] procedure.Procedure(306): Waiting on: java.util.concurrent.CountDownLatch@521c38c3[Count = 0] remaining members to acquire global barrier 2015-04-04 10:48:02,689 DEBUG [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] procedure.ZKProcedureCoordinatorRpcs(115): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestCompactionOverride 2015-04-04 10:48:02,690 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestCompactionOverride 2015-04-04 10:48:02,690 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/TestCompactionOverride 2015-04-04 10:48:02,690 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Recieved reached global barrier:/hbase/flush-table-proc/reached/TestCompactionOverride 2015-04-04 10:48:02,690 DEBUG [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] zookeeper.ZKUtil(492): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestCompactionOverride/localhost,51330,1428169644428 2015-04-04 10:48:02,690 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] procedure.Subprocedure(181): Subprocedure 'TestCompactionOverride' received 'reached' from coordinator. 2015-04-04 10:48:02,690 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] procedure.Subprocedure(183): Subprocedure 'TestCompactionOverride' locally completed 2015-04-04 10:48:02,690 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'TestCompactionOverride' completed for member 'localhost,51330,1428169644428' in zk 2015-04-04 10:48:02,692 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestCompactionOverride/localhost,51330,1428169644428 2015-04-04 10:48:02,692 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(186): Node created: /hbase/flush-table-proc/reached/TestCompactionOverride/localhost,51330,1428169644428 2015-04-04 10:48:02,692 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2015-04-04 10:48:02,692 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2015-04-04 10:48:02,692 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] procedure.Subprocedure(188): Subprocedure 'TestCompactionOverride' has notified controller of completion 2015-04-04 10:48:02,692 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2015-04-04 10:48:02,692 DEBUG [member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1] procedure.Subprocedure(213): Subprocedure 'TestCompactionOverride' completed. 2015-04-04 10:48:02,693 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2015-04-04 10:48:02,693 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestCompactionOverride 2015-04-04 10:48:02,694 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------localhost,51330,1428169644428 2015-04-04 10:48:02,694 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2015-04-04 10:48:02,694 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2015-04-04 10:48:02,695 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestCompactionOverride 2015-04-04 10:48:02,695 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------localhost,51330,1428169644428 2015-04-04 10:48:02,696 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(211): Finished data from procedure 'TestCompactionOverride' member 'localhost,51330,1428169644428': 2015-04-04 10:48:02,696 INFO [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] procedure.Procedure(220): Procedure 'TestCompactionOverride' execution completed 2015-04-04 10:48:02,696 DEBUG [main-EventThread] procedure.Procedure(328): Member: 'localhost,51330,1428169644428' released barrier for procedure'TestCompactionOverride', counting down latch. Waiting for 0 more 2015-04-04 10:48:02,696 DEBUG [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] procedure.Procedure(229): Running finish phase. 2015-04-04 10:48:02,696 DEBUG [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] procedure.Procedure(280): Finished coordinator procedure - removing self from list of running procedures 2015-04-04 10:48:02,696 DEBUG [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] procedure.ZKProcedureCoordinatorRpcs(157): Attempting to clean out zk node for op:TestCompactionOverride 2015-04-04 10:48:02,696 INFO [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] procedure.ZKProcedureUtil(284): Clearing all znodes for procedure TestCompactionOverrideincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2015-04-04 10:48:02,698 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestCompactionOverride/localhost,51330,1428169644428 2015-04-04 10:48:02,699 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestCompactionOverride 2015-04-04 10:48:02,699 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2015-04-04 10:48:02,699 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2015-04-04 10:48:02,699 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2015-04-04 10:48:02,700 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestCompactionOverride/localhost,51330,1428169644428 2015-04-04 10:48:02,701 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestCompactionOverride 2015-04-04 10:48:02,702 DEBUG [(localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2015-04-04 10:48:02,702 INFO [B.defaultRpcServer.handler=0,queue=0,port=51327] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'TestCompactionOverride' 2015-04-04 10:48:02,702 INFO [B.defaultRpcServer.handler=0,queue=0,port=51327] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2015-04-04 10:48:02,704 DEBUG [Thread-448] client.HBaseAdmin(3381): Waiting a max of 60000 ms for procedure 'flush-table-proc : TestCompactionOverride'' to complete. (max 1714 ms per retry) 2015-04-04 10:48:02,704 DEBUG [Thread-448] client.HBaseAdmin(3390): (#1) Sleeping: 100ms while waiting for procedure completion. 2015-04-04 10:48:02,804 DEBUG [Thread-448] client.HBaseAdmin(3396): Getting current status of procedure from master... 2015-04-04 10:48:02,820 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51327] master.MasterRpcServices(883): Checking to see if procedure from request:flush-table-proc is done 2015-04-04 10:48:02,822 DEBUG [Thread-448] coprocessor.TestRegionObserverInterface(532): Flush complete 2015-04-04 10:48:02,829 INFO [Thread-448] zookeeper.RecoverableZooKeeper(121): Process identifier=hbase-admin-on-hconnection-0x3b4ad720 connecting to ZooKeeper ensemble=localhost:64763 2015-04-04 10:48:02,832 DEBUG [Thread-448-EventThread] zookeeper.ZooKeeperWatcher(401): hbase-admin-on-hconnection-0x3b4ad7200x0, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2015-04-04 10:48:02,834 DEBUG [Thread-448-EventThread] zookeeper.ZooKeeperWatcher(485): hbase-admin-on-hconnection-0x3b4ad720-0x14c858b55e6001a connected 2015-04-04 10:48:02,837 DEBUG [Thread-448] ipc.RpcClientImpl$Connection(343): Use SIMPLE authentication for service AdminService, sasl=false 2015-04-04 10:48:02,838 DEBUG [Thread-448] ipc.RpcClientImpl$Connection(708): Connecting to localhost/127.0.0.1:51330 2015-04-04 10:48:02,838 DEBUG [RpcServer.listener,port=51330] ipc.RpcServer$Listener(829): RpcServer.listener,port=51330: connection from 127.0.0.1:51449; # active connections: 6 2015-04-04 10:48:02,839 INFO [RpcServer.reader=0,bindAddress=localhost,port=51330] ipc.RpcServer$Connection(1661): Connection from 127.0.0.1 port: 51449 with version info: version: "1.1.0-SNAPSHOT" url: "git://apurtell-ltm2.internal.salesforce.com/Users/apurtell/src/hbase" revision: "5c19f9eac7d1f22147345f09aad2f04fec1ad67d" user: "apurtell" date: "Sat Apr 4 10:46:42 PDT 2015" src_checksum: "2a921de567e8e6d9e1c358f3276c1ec9" 2015-04-04 10:48:02,839 INFO [PriorityRpcServer.handler=5,queue=1,port=51330] regionserver.RSRpcServices(1046): Compacting TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. 2015-04-04 10:48:02,840 DEBUG [PriorityRpcServer.handler=5,queue=1,port=51330] compactions.RatioBasedCompactionPolicy(92): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 10 blocking 2015-04-04 10:48:02,842 DEBUG [PriorityRpcServer.handler=5,queue=1,port=51330] regionserver.HStore(1578): 1d8e5c1a351aa112c824cf548d63bb46 - a: Initiating major compaction (all files) 2015-04-04 10:48:02,844 DEBUG [PriorityRpcServer.handler=5,queue=1,port=51330] regionserver.CompactSplitThread(355): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@12ac0e87; Because: User-triggered major compaction; compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-04-04 10:48:02,844 INFO [RS:0;localhost:51330-shortCompactions-1428169682844] regionserver.HRegion(1770): Starting compaction on a in region TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. 2015-04-04 10:48:02,845 INFO [RS:0;localhost:51330-shortCompactions-1428169682844] regionserver.HStore(1206): Starting compaction of 1 file(s) in a of TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. into tmpdir=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestCompactionOverride/1d8e5c1a351aa112c824cf548d63bb46/.tmp, totalSize=5.2 K 2015-04-04 10:48:02,847 DEBUG [RS:0;localhost:51330-shortCompactions-1428169682844] compactions.Compactor(169): Compacting hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestCompactionOverride/1d8e5c1a351aa112c824cf548d63bb46/a/c8545aca9121486d9414ba50a6613cd4, keycount=10, bloomtype=ROW, size=5.2 K, encoding=NONE, seqNum=24, earliestPutTs=1428169682597 2015-04-04 10:48:02,859 INFO [RS:0;localhost:51330-shortCompactions-1428169682844] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=0, currentSize=1071760, freeSize=1042890544, maxSize=1043962304, heapSize=1071760, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:02,870 INFO [IPC Server handler 2 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741868_1044{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 5084 2015-04-04 10:48:03,286 DEBUG [RS:0;localhost:51330-shortCompactions-1428169682844] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestCompactionOverride/1d8e5c1a351aa112c824cf548d63bb46/.tmp/e4a4f181ef044d0089eb05bc03c91de3 as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestCompactionOverride/1d8e5c1a351aa112c824cf548d63bb46/a/e4a4f181ef044d0089eb05bc03c91de3 2015-04-04 10:48:03,308 DEBUG [RS:0;localhost:51330-shortCompactions-1428169682844] regionserver.HStore(1713): Removing store files after compaction... 2015-04-04 10:48:03,311 DEBUG [RS:0;localhost:51330-shortCompactions-1428169682844] backup.HFileArchiver(224): Archiving compacted store files. 2015-04-04 10:48:03,316 DEBUG [RS:0;localhost:51330-shortCompactions-1428169682844] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestCompactionOverride/1d8e5c1a351aa112c824cf548d63bb46/a/c8545aca9121486d9414ba50a6613cd4, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestCompactionOverride/1d8e5c1a351aa112c824cf548d63bb46/a/c8545aca9121486d9414ba50a6613cd4 2015-04-04 10:48:03,316 INFO [RS:0;localhost:51330-shortCompactions-1428169682844] regionserver.HStore(1334): Completed major compaction of 1 (all) file(s) in a of TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. into e4a4f181ef044d0089eb05bc03c91de3(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2015-04-04 10:48:03,318 INFO [RS:0;localhost:51330-shortCompactions-1428169682844] regionserver.CompactSplitThread$CompactionRunner(523): Completed compaction: Request = regionName=TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46., storeName=a, fileCount=1, fileSize=5.2 K, priority=1, time=1428169682841874000; duration=0sec 2015-04-04 10:48:03,319 DEBUG [RS:0;localhost:51330-shortCompactions-1428169682844] regionserver.CompactSplitThread$CompactionRunner(545): CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2015-04-04 10:48:03,850 DEBUG [Thread-448] coprocessor.TestRegionObserverInterface(543): Last compaction was at 1428169683294 2015-04-04 10:48:03,897 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testCompactionOverride Thread=479 (was 457) Potentially hanging thread: RS:0;localhost:51330-shortCompactions-1428169682844 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:539) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t104 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t77 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x34c6ce8e-shared--pool9-t12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51391 [Waiting for operation #44] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51416 [Waiting for operation #21] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: (localhost,51327,1428169644129)-proc-coordinator-pool3-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t105 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t82 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t106 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t78 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t107 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t76 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t80 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51417 [Waiting for operation #17] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t72 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: member: 'localhost,51330,1428169644428' subprocedure-pool4-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t73 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t71 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t108 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t81 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t74 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t79 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t75 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x130f424b-metaLookup-shared--pool77-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=439 (was 439), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=217 (was 217), ProcessCount=232 (was 232), AvailableMemoryMB=18 (was 18), ConnectionCount=1 (was 1) 2015-04-04 10:48:03,926 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testCheckAndPutHooks Thread=479, OpenFileDescriptor=439, MaxFileDescriptor=10240, SystemLoadAverage=217, ProcessCount=232, AvailableMemoryMB=18, ConnectionCount=1 2015-04-04 10:48:03,929 INFO [B.defaultRpcServer.handler=1,queue=0,port=51327] master.HMaster(1318): Client=apurtell/null create 'TestTable.testCheckAndPutHooks', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:48:03,933 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testCheckAndPutHooks/write-master:513270000000000 2015-04-04 10:48:03,941 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(188): Create table TestTable.testCheckAndPutHooks 2015-04-04 10:48:03,951 INFO [IPC Server handler 3 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741869_1045{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 794 2015-04-04 10:48:04,356 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testCheckAndPutHooks/.tabledesc/.tableinfo.0000000001 2015-04-04 10:48:04,359 INFO [RegionOpenAndInitThread-TestTable.testCheckAndPutHooks-1] regionserver.HRegion(5885): creating HRegion TestTable.testCheckAndPutHooks HTD == 'TestTable.testCheckAndPutHooks', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp Table name == TestTable.testCheckAndPutHooks 2015-04-04 10:48:04,369 INFO [IPC Server handler 4 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741870_1046{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 65 2015-04-04 10:48:04,776 DEBUG [RegionOpenAndInitThread-TestTable.testCheckAndPutHooks-1] regionserver.HRegion(721): Instantiated TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. 2015-04-04 10:48:04,776 DEBUG [RegionOpenAndInitThread-TestTable.testCheckAndPutHooks-1] regionserver.HRegion(1350): Closing TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5.: disabling compactions & flushes 2015-04-04 10:48:04,776 DEBUG [RegionOpenAndInitThread-TestTable.testCheckAndPutHooks-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. 2015-04-04 10:48:04,776 INFO [RegionOpenAndInitThread-TestTable.testCheckAndPutHooks-1] regionserver.HRegion(1471): Closed TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. 2015-04-04 10:48:04,782 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:48:04,782 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,51330,1428169644428 2015-04-04 10:48:04,783 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKAssign(174): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Async create of unassigned node 628c398fc1fdb855de25fb3e58bea4a5 with OFFLINE state 2015-04-04 10:48:04,792 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:04,792 DEBUG [main-EventThread] master.OfflineCallback(69): rs={628c398fc1fdb855de25fb3e58bea4a5 state=OFFLINE, ts=1428169684782, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:48:04,793 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={628c398fc1fdb855de25fb3e58bea4a5 state=OFFLINE, ts=1428169684782, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:48:04,796 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1658): localhost,51330,1428169644428 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:48:04,796 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.RegionStates(1112): Transition {628c398fc1fdb855de25fb3e58bea4a5 state=OFFLINE, ts=1428169684783, server=null} to {628c398fc1fdb855de25fb3e58bea4a5 state=PENDING_OPEN, ts=1428169684796, server=localhost,51330,1428169644428} 2015-04-04 10:48:04,797 INFO [PriorityRpcServer.handler=5,queue=1,port=51330] regionserver.RSRpcServices(1390): Open TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. 2015-04-04 10:48:04,803 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning 628c398fc1fdb855de25fb3e58bea4a5 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:48:04,803 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1788): Bulk assigning done for localhost,51330,1428169644428 2015-04-04 10:48:04,804 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testCheckAndPutHooks state from ENABLING to ENABLED 2015-04-04 10:48:04,805 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/628c398fc1fdb855de25fb3e58bea4a5 2015-04-04 10:48:04,805 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node 628c398fc1fdb855de25fb3e58bea4a5 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:48:04,806 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(6078): Opening region: {ENCODED => 628c398fc1fdb855de25fb3e58bea4a5, NAME => 'TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:48:04,806 DEBUG [AM.ZK.Worker-pool3-t15] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,51330,1428169644428, region=628c398fc1fdb855de25fb3e58bea4a5, current_state={628c398fc1fdb855de25fb3e58bea4a5 state=PENDING_OPEN, ts=1428169684796, server=localhost,51330,1428169644428} 2015-04-04 10:48:04,806 INFO [AM.ZK.Worker-pool3-t15] master.RegionStates(1112): Transition {628c398fc1fdb855de25fb3e58bea4a5 state=PENDING_OPEN, ts=1428169684796, server=localhost,51330,1428169644428} to {628c398fc1fdb855de25fb3e58bea4a5 state=OPENING, ts=1428169684806, server=localhost,51330,1428169644428} 2015-04-04 10:48:04,806 DEBUG [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:48:04,806 DEBUG [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:48:04,806 INFO [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:48:04,807 INFO [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:48:04,807 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testCheckAndPutHooks 628c398fc1fdb855de25fb3e58bea4a5 2015-04-04 10:48:04,807 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(721): Instantiated TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. 2015-04-04 10:48:04,810 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testCheckAndPutHooks/write-master:513270000000000 2015-04-04 10:48:04,810 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(218): Table, TestTable.testCheckAndPutHooks, creation successful 2015-04-04 10:48:04,811 INFO [StoreOpener-628c398fc1fdb855de25fb3e58bea4a5-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:04,811 INFO [StoreOpener-628c398fc1fdb855de25fb3e58bea4a5-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:04,812 DEBUG [StoreOpener-628c398fc1fdb855de25fb3e58bea4a5-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/a 2015-04-04 10:48:04,814 INFO [StoreOpener-628c398fc1fdb855de25fb3e58bea4a5-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:04,814 INFO [StoreOpener-628c398fc1fdb855de25fb3e58bea4a5-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:04,815 DEBUG [StoreOpener-628c398fc1fdb855de25fb3e58bea4a5-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/b 2015-04-04 10:48:04,817 INFO [StoreOpener-628c398fc1fdb855de25fb3e58bea4a5-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:04,818 INFO [StoreOpener-628c398fc1fdb855de25fb3e58bea4a5-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:04,819 DEBUG [StoreOpener-628c398fc1fdb855de25fb3e58bea4a5-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/c 2015-04-04 10:48:04,820 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5 2015-04-04 10:48:04,822 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:48:04,826 DEBUG [RS_OPEN_REGION-localhost:51330-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:48:04,826 INFO [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(847): Onlined 628c398fc1fdb855de25fb3e58bea4a5; next sequenceid=2 2015-04-04 10:48:04,827 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(643): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Attempting to retransition opening state of node 628c398fc1fdb855de25fb3e58bea4a5 2015-04-04 10:48:04,828 INFO [PostOpenDeployTasks:628c398fc1fdb855de25fb3e58bea4a5] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. 2015-04-04 10:48:04,831 INFO [PostOpenDeployTasks:628c398fc1fdb855de25fb3e58bea4a5] hbase.MetaTableAccessor(1350): Updated row TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. with server=localhost,51330,1428169644428 2015-04-04 10:48:04,831 DEBUG [PostOpenDeployTasks:628c398fc1fdb855de25fb3e58bea4a5] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. 2015-04-04 10:48:04,831 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning 628c398fc1fdb855de25fb3e58bea4a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:48:04,833 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/628c398fc1fdb855de25fb3e58bea4a5 2015-04-04 10:48:04,833 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node 628c398fc1fdb855de25fb3e58bea4a5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:48:04,833 DEBUG [RS_OPEN_REGION-localhost:51330-0] coordination.ZkOpenRegionCoordination(91): Transitioned 628c398fc1fdb855de25fb3e58bea4a5 to OPENED in zk on localhost,51330,1428169644428 2015-04-04 10:48:04,833 DEBUG [RS_OPEN_REGION-localhost:51330-0] handler.OpenRegionHandler(167): Opened TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. on localhost,51330,1428169644428 2015-04-04 10:48:04,833 DEBUG [AM.ZK.Worker-pool3-t16] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,51330,1428169644428, region=628c398fc1fdb855de25fb3e58bea4a5, current_state={628c398fc1fdb855de25fb3e58bea4a5 state=OPENING, ts=1428169684806, server=localhost,51330,1428169644428} 2015-04-04 10:48:04,833 INFO [AM.ZK.Worker-pool3-t16] master.RegionStates(1112): Transition {628c398fc1fdb855de25fb3e58bea4a5 state=OPENING, ts=1428169684806, server=localhost,51330,1428169644428} to {628c398fc1fdb855de25fb3e58bea4a5 state=OPEN, ts=1428169684833, server=localhost,51330,1428169644428} 2015-04-04 10:48:04,834 DEBUG [AM.ZK.Worker-pool3-t16] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 628c398fc1fdb855de25fb3e58bea4a5 from localhost,51327,1428169644129; deleting unassigned node 2015-04-04 10:48:04,835 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/628c398fc1fdb855de25fb3e58bea4a5 2015-04-04 10:48:04,835 DEBUG [AM.ZK.Worker-pool3-t16] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node 628c398fc1fdb855de25fb3e58bea4a5 in expected state RS_ZK_REGION_OPENED 2015-04-04 10:48:04,835 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:04,835 DEBUG [AM.ZK.Worker-pool3-t16] master.AssignmentManager$4(1337): Znode TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. deleted, state: {628c398fc1fdb855de25fb3e58bea4a5 state=OPEN, ts=1428169684833, server=localhost,51330,1428169644428} 2015-04-04 10:48:04,835 DEBUG [AM.ZK.Worker-pool3-t16] master.RegionStates(460): Onlined 628c398fc1fdb855de25fb3e58bea4a5 on localhost,51330,1428169644428 {ENCODED => 628c398fc1fdb855de25fb3e58bea4a5, NAME => 'TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:48:05,096 INFO [Thread-470] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:48:05,101 INFO [Thread-470] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:48:05,118 INFO [Thread-470] client.HBaseAdmin$9(979): Started disable of TestTable.testCheckAndPutHooks 2015-04-04 10:48:05,119 INFO [B.defaultRpcServer.handler=4,queue=0,port=51327] master.HMaster(1659): Client=apurtell/null disable TestTable.testCheckAndPutHooks 2015-04-04 10:48:05,120 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testCheckAndPutHooks/write-master:513270000000001 2015-04-04 10:48:05,126 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(129): Attempting to disable table TestTable.testCheckAndPutHooks 2015-04-04 10:48:05,126 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testCheckAndPutHooks state from DISABLING to DISABLING 2015-04-04 10:48:05,127 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:48:05,128 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. (offlining), current state: {628c398fc1fdb855de25fb3e58bea4a5 state=OPEN, ts=1428169684835, server=localhost,51330,1428169644428} 2015-04-04 10:48:05,128 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Creating unassigned node 628c398fc1fdb855de25fb3e58bea4a5 in a CLOSING state 2015-04-04 10:48:05,129 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:05,129 INFO [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {628c398fc1fdb855de25fb3e58bea4a5 state=OPEN, ts=1428169684835, server=localhost,51330,1428169644428} to {628c398fc1fdb855de25fb3e58bea4a5 state=PENDING_CLOSE, ts=1428169685129, server=localhost,51330,1428169644428} 2015-04-04 10:48:05,130 INFO [PriorityRpcServer.handler=9,queue=1,port=51330] regionserver.RSRpcServices(1018): Close 628c398fc1fdb855de25fb3e58bea4a5, moving to null 2015-04-04 10:48:05,130 DEBUG [RS_CLOSE_REGION-localhost:51330-1] handler.CloseRegionHandler(119): Processing close of TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. 2015-04-04 10:48:05,130 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,51330,1428169644428 for region TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. 2015-04-04 10:48:05,131 DEBUG [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(1350): Closing TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5.: disabling compactions & flushes 2015-04-04 10:48:05,131 DEBUG [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. 2015-04-04 10:48:05,132 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(2084): Started memstore flush for TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5., current region memstore size 320 B, and 3/3 column families' memstores are being flushed. 2015-04-04 10:48:05,132 DEBUG [Thread-470] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in TestTable.testCheckAndPutHooks 2015-04-04 10:48:05,142 INFO [IPC Server handler 4 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741871_1047{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 4718 2015-04-04 10:48:05,243 DEBUG [Thread-470] client.HBaseAdmin(1019): Sleeping= 200ms, waiting for all regions to be disabled in TestTable.testCheckAndPutHooks 2015-04-04 10:48:05,450 DEBUG [Thread-470] client.HBaseAdmin(1019): Sleeping= 300ms, waiting for all regions to be disabled in TestTable.testCheckAndPutHooks 2015-04-04 10:48:05,546 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/.tmp/7a77e7fbcefa404199a0eb6b366d6f06 2015-04-04 10:48:05,556 DEBUG [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/.tmp/7a77e7fbcefa404199a0eb6b366d6f06 as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/a/7a77e7fbcefa404199a0eb6b366d6f06 2015-04-04 10:48:05,564 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/a/7a77e7fbcefa404199a0eb6b366d6f06, entries=1, sequenceid=8, filesize=4.6 K 2015-04-04 10:48:05,565 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(2363): Finished memstore flush of ~320 B/320, currentsize=0 B/0 for region TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. in 434ms, sequenceid=8, compaction requested=false 2015-04-04 10:48:05,567 INFO [StoreCloserThread-TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5.-1] regionserver.HStore(879): Closed a 2015-04-04 10:48:05,567 INFO [StoreCloserThread-TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5.-1] regionserver.HStore(879): Closed b 2015-04-04 10:48:05,567 INFO [StoreCloserThread-TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5.-1] regionserver.HStore(879): Closed c 2015-04-04 10:48:05,574 DEBUG [RS_CLOSE_REGION-localhost:51330-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/recovered.edits/11.seqid to file, newSeqId=11, maxSeqId=2 2015-04-04 10:48:05,575 DEBUG [RS_CLOSE_REGION-localhost:51330-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:48:05,575 DEBUG [RS_CLOSE_REGION-localhost:51330-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:48:05,575 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(1471): Closed TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. 2015-04-04 10:48:05,576 DEBUG [RS_CLOSE_REGION-localhost:51330-1] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning 628c398fc1fdb855de25fb3e58bea4a5 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:48:05,585 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/628c398fc1fdb855de25fb3e58bea4a5 2015-04-04 10:48:05,585 DEBUG [RS_CLOSE_REGION-localhost:51330-1] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node 628c398fc1fdb855de25fb3e58bea4a5 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:48:05,586 DEBUG [RS_CLOSE_REGION-localhost:51330-1] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. on localhost,51330,1428169644428 2015-04-04 10:48:05,586 DEBUG [RS_CLOSE_REGION-localhost:51330-1] handler.CloseRegionHandler(164): Closed TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. 2015-04-04 10:48:05,586 DEBUG [AM.ZK.Worker-pool3-t1] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,51330,1428169644428, region=628c398fc1fdb855de25fb3e58bea4a5, current_state={628c398fc1fdb855de25fb3e58bea4a5 state=PENDING_CLOSE, ts=1428169685129, server=localhost,51330,1428169644428} 2015-04-04 10:48:05,586 DEBUG [AM.ZK.Worker-pool3-t1] handler.ClosedRegionHandler(93): Handling CLOSED event for 628c398fc1fdb855de25fb3e58bea4a5 2015-04-04 10:48:05,587 DEBUG [AM.ZK.Worker-pool3-t1] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. 2015-04-04 10:48:05,588 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/628c398fc1fdb855de25fb3e58bea4a5 2015-04-04 10:48:05,588 DEBUG [AM.ZK.Worker-pool3-t1] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node 628c398fc1fdb855de25fb3e58bea4a5 in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:48:05,588 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:05,589 INFO [AM.ZK.Worker-pool3-t1] master.RegionStates(1112): Transition {628c398fc1fdb855de25fb3e58bea4a5 state=PENDING_CLOSE, ts=1428169685129, server=localhost,51330,1428169644428} to {628c398fc1fdb855de25fb3e58bea4a5 state=OFFLINE, ts=1428169685589, server=localhost,51330,1428169644428} 2015-04-04 10:48:05,589 INFO [AM.ZK.Worker-pool3-t1] master.RegionStates(600): Offlined 628c398fc1fdb855de25fb3e58bea4a5 from localhost,51330,1428169644428 2015-04-04 10:48:05,604 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testCheckAndPutHooks state from DISABLING to DISABLED 2015-04-04 10:48:05,606 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(191): Disabled table, TestTable.testCheckAndPutHooks, is done=true 2015-04-04 10:48:05,608 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testCheckAndPutHooks/write-master:513270000000001 2015-04-04 10:48:05,761 INFO [Thread-470] client.HBaseAdmin(1034): Disabled TestTable.testCheckAndPutHooks 2015-04-04 10:48:05,762 INFO [B.defaultRpcServer.handler=1,queue=0,port=51327] master.HMaster(1564): Client=apurtell/null delete TestTable.testCheckAndPutHooks 2015-04-04 10:48:05,773 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testCheckAndPutHooks/write-master:513270000000002 2015-04-04 10:48:05,779 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table TestTable.testCheckAndPutHooks 2015-04-04 10:48:05,784 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:48:05,787 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => 628c398fc1fdb855de25fb3e58bea4a5, NAME => 'TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:48:05,789 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(186): Archiving region TestTable.testCheckAndPutHooks,,1428169683929.628c398fc1fdb855de25fb3e58bea4a5. from FS 2015-04-04 10:48:05,789 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5 2015-04-04 10:48:05,794 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/recovered.edits] 2015-04-04 10:48:05,802 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/a/7a77e7fbcefa404199a0eb6b366d6f06, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/a/7a77e7fbcefa404199a0eb6b366d6f06 2015-04-04 10:48:05,813 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/recovered.edits/11.seqid, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5/recovered.edits/11.seqid 2015-04-04 10:48:05,814 INFO [IPC Server handler 2 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741870_1046 127.0.0.1:51320 2015-04-04 10:48:05,814 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testCheckAndPutHooks/628c398fc1fdb855de25fb3e58bea4a5 2015-04-04 10:48:05,815 INFO [IPC Server handler 8 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741869_1045 127.0.0.1:51320 2015-04-04 10:48:05,815 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(196): Table 'TestTable.testCheckAndPutHooks' archived! 2015-04-04 10:48:05,815 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(116): Removing 'TestTable.testCheckAndPutHooks' descriptor. 2015-04-04 10:48:05,816 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(122): Removing 'TestTable.testCheckAndPutHooks' from region states. 2015-04-04 10:48:05,816 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(126): Marking 'TestTable.testCheckAndPutHooks' as deleted. 2015-04-04 10:48:05,822 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testCheckAndPutHooks/write-master:513270000000002 2015-04-04 10:48:05,893 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51327] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = TestTable.testCheckAndPutHooks org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testCheckAndPutHooks at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:48:05,893 INFO [Thread-470] client.HBaseAdmin(739): Deleted TestTable.testCheckAndPutHooks 2015-04-04 10:48:05,923 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testCheckAndPutHooks Thread=503 (was 479) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t113 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t90 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51416 [Waiting for operation #24] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t109 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t110 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t119 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t87 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t86 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t91 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x34c6ce8e-shared--pool9-t13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t84 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t111 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t88 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t118 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t92 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1796537615_1 at /127.0.0.1:51417 [Waiting for operation #20] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t93 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t115 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t114 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t85 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t83 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t116 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t112 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t117 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t120 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t89 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=439 (was 439), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=217 (was 217), ProcessCount=232 (was 232), AvailableMemoryMB=17 (was 18), ConnectionCount=1 (was 1) 2015-04-04 10:48:05,924 WARN [main] hbase.ResourceChecker(134): Thread=503 is superior to 500 2015-04-04 10:48:05,960 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testRowMutation Thread=503, OpenFileDescriptor=439, MaxFileDescriptor=10240, SystemLoadAverage=217, ProcessCount=232, AvailableMemoryMB=17, ConnectionCount=1 2015-04-04 10:48:05,960 WARN [main] hbase.ResourceChecker(134): Thread=503 is superior to 500 2015-04-04 10:48:05,962 INFO [B.defaultRpcServer.handler=1,queue=0,port=51327] master.HMaster(1318): Client=apurtell/null create 'TestTable.testRowMutation', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:48:05,975 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testRowMutation/write-master:513270000000000 2015-04-04 10:48:05,983 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(188): Create table TestTable.testRowMutation 2015-04-04 10:48:05,993 INFO [IPC Server handler 0 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741872_1048{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 0 2015-04-04 10:48:05,996 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRowMutation/.tabledesc/.tableinfo.0000000001 2015-04-04 10:48:05,997 INFO [RegionOpenAndInitThread-TestTable.testRowMutation-1] regionserver.HRegion(5885): creating HRegion TestTable.testRowMutation HTD == 'TestTable.testRowMutation', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp Table name == TestTable.testRowMutation 2015-04-04 10:48:06,011 INFO [IPC Server handler 0 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741873_1049{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 0 2015-04-04 10:48:06,012 DEBUG [RegionOpenAndInitThread-TestTable.testRowMutation-1] regionserver.HRegion(721): Instantiated TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. 2015-04-04 10:48:06,012 DEBUG [RegionOpenAndInitThread-TestTable.testRowMutation-1] regionserver.HRegion(1350): Closing TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2.: disabling compactions & flushes 2015-04-04 10:48:06,012 DEBUG [RegionOpenAndInitThread-TestTable.testRowMutation-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. 2015-04-04 10:48:06,012 INFO [RegionOpenAndInitThread-TestTable.testRowMutation-1] regionserver.HRegion(1471): Closed TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. 2015-04-04 10:48:06,016 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:48:06,017 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,51330,1428169644428 2015-04-04 10:48:06,017 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKAssign(174): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Async create of unassigned node 7cd74fdda353b2622f22dc40ab9c09f2 with OFFLINE state 2015-04-04 10:48:06,018 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:06,018 DEBUG [main-EventThread] master.OfflineCallback(69): rs={7cd74fdda353b2622f22dc40ab9c09f2 state=OFFLINE, ts=1428169686017, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:48:06,019 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={7cd74fdda353b2622f22dc40ab9c09f2 state=OFFLINE, ts=1428169686017, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:48:06,023 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1658): localhost,51330,1428169644428 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:48:06,023 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.RegionStates(1112): Transition {7cd74fdda353b2622f22dc40ab9c09f2 state=OFFLINE, ts=1428169686017, server=null} to {7cd74fdda353b2622f22dc40ab9c09f2 state=PENDING_OPEN, ts=1428169686023, server=localhost,51330,1428169644428} 2015-04-04 10:48:06,024 INFO [PriorityRpcServer.handler=6,queue=0,port=51330] regionserver.RSRpcServices(1390): Open TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. 2015-04-04 10:48:06,030 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning 7cd74fdda353b2622f22dc40ab9c09f2 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:48:06,030 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1788): Bulk assigning done for localhost,51330,1428169644428 2015-04-04 10:48:06,030 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testRowMutation state from ENABLING to ENABLED 2015-04-04 10:48:06,032 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/7cd74fdda353b2622f22dc40ab9c09f2 2015-04-04 10:48:06,032 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node 7cd74fdda353b2622f22dc40ab9c09f2 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:48:06,032 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(6078): Opening region: {ENCODED => 7cd74fdda353b2622f22dc40ab9c09f2, NAME => 'TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:48:06,033 DEBUG [AM.ZK.Worker-pool3-t5] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,51330,1428169644428, region=7cd74fdda353b2622f22dc40ab9c09f2, current_state={7cd74fdda353b2622f22dc40ab9c09f2 state=PENDING_OPEN, ts=1428169686023, server=localhost,51330,1428169644428} 2015-04-04 10:48:06,033 INFO [AM.ZK.Worker-pool3-t5] master.RegionStates(1112): Transition {7cd74fdda353b2622f22dc40ab9c09f2 state=PENDING_OPEN, ts=1428169686023, server=localhost,51330,1428169644428} to {7cd74fdda353b2622f22dc40ab9c09f2 state=OPENING, ts=1428169686033, server=localhost,51330,1428169644428} 2015-04-04 10:48:06,033 DEBUG [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:48:06,033 DEBUG [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:48:06,033 INFO [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:48:06,033 INFO [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:48:06,034 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testRowMutation 7cd74fdda353b2622f22dc40ab9c09f2 2015-04-04 10:48:06,034 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(721): Instantiated TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. 2015-04-04 10:48:06,037 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testRowMutation/write-master:513270000000000 2015-04-04 10:48:06,037 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(218): Table, TestTable.testRowMutation, creation successful 2015-04-04 10:48:06,038 INFO [StoreOpener-7cd74fdda353b2622f22dc40ab9c09f2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:06,038 INFO [StoreOpener-7cd74fdda353b2622f22dc40ab9c09f2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:06,039 DEBUG [StoreOpener-7cd74fdda353b2622f22dc40ab9c09f2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/a 2015-04-04 10:48:06,041 INFO [StoreOpener-7cd74fdda353b2622f22dc40ab9c09f2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:06,041 INFO [StoreOpener-7cd74fdda353b2622f22dc40ab9c09f2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:06,042 DEBUG [StoreOpener-7cd74fdda353b2622f22dc40ab9c09f2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/b 2015-04-04 10:48:06,044 INFO [StoreOpener-7cd74fdda353b2622f22dc40ab9c09f2-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:06,044 INFO [StoreOpener-7cd74fdda353b2622f22dc40ab9c09f2-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:06,045 DEBUG [StoreOpener-7cd74fdda353b2622f22dc40ab9c09f2-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/c 2015-04-04 10:48:06,046 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2 2015-04-04 10:48:06,048 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:48:06,052 DEBUG [RS_OPEN_REGION-localhost:51330-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:48:06,052 INFO [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(847): Onlined 7cd74fdda353b2622f22dc40ab9c09f2; next sequenceid=2 2015-04-04 10:48:06,054 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(643): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Attempting to retransition opening state of node 7cd74fdda353b2622f22dc40ab9c09f2 2015-04-04 10:48:06,055 INFO [PostOpenDeployTasks:7cd74fdda353b2622f22dc40ab9c09f2] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. 2015-04-04 10:48:06,057 INFO [PostOpenDeployTasks:7cd74fdda353b2622f22dc40ab9c09f2] hbase.MetaTableAccessor(1350): Updated row TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. with server=localhost,51330,1428169644428 2015-04-04 10:48:06,058 DEBUG [PostOpenDeployTasks:7cd74fdda353b2622f22dc40ab9c09f2] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. 2015-04-04 10:48:06,058 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning 7cd74fdda353b2622f22dc40ab9c09f2 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:48:06,060 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node 7cd74fdda353b2622f22dc40ab9c09f2 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:48:06,060 DEBUG [RS_OPEN_REGION-localhost:51330-1] coordination.ZkOpenRegionCoordination(91): Transitioned 7cd74fdda353b2622f22dc40ab9c09f2 to OPENED in zk on localhost,51330,1428169644428 2015-04-04 10:48:06,060 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/7cd74fdda353b2622f22dc40ab9c09f2 2015-04-04 10:48:06,060 DEBUG [RS_OPEN_REGION-localhost:51330-1] handler.OpenRegionHandler(167): Opened TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. on localhost,51330,1428169644428 2015-04-04 10:48:06,061 DEBUG [AM.ZK.Worker-pool3-t20] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,51330,1428169644428, region=7cd74fdda353b2622f22dc40ab9c09f2, current_state={7cd74fdda353b2622f22dc40ab9c09f2 state=OPENING, ts=1428169686033, server=localhost,51330,1428169644428} 2015-04-04 10:48:06,061 INFO [AM.ZK.Worker-pool3-t20] master.RegionStates(1112): Transition {7cd74fdda353b2622f22dc40ab9c09f2 state=OPENING, ts=1428169686033, server=localhost,51330,1428169644428} to {7cd74fdda353b2622f22dc40ab9c09f2 state=OPEN, ts=1428169686061, server=localhost,51330,1428169644428} 2015-04-04 10:48:06,061 DEBUG [AM.ZK.Worker-pool3-t20] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 7cd74fdda353b2622f22dc40ab9c09f2 from localhost,51327,1428169644129; deleting unassigned node 2015-04-04 10:48:06,062 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/7cd74fdda353b2622f22dc40ab9c09f2 2015-04-04 10:48:06,062 DEBUG [AM.ZK.Worker-pool3-t20] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node 7cd74fdda353b2622f22dc40ab9c09f2 in expected state RS_ZK_REGION_OPENED 2015-04-04 10:48:06,062 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:06,062 DEBUG [AM.ZK.Worker-pool3-t20] master.AssignmentManager$4(1337): Znode TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. deleted, state: {7cd74fdda353b2622f22dc40ab9c09f2 state=OPEN, ts=1428169686061, server=localhost,51330,1428169644428} 2015-04-04 10:48:06,063 DEBUG [AM.ZK.Worker-pool3-t20] master.RegionStates(460): Onlined 7cd74fdda353b2622f22dc40ab9c09f2 on localhost,51330,1428169644428 {ENCODED => 7cd74fdda353b2622f22dc40ab9c09f2, NAME => 'TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:48:06,103 INFO [Thread-485] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:48:06,107 INFO [Thread-485] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:48:06,122 INFO [Thread-485] client.HBaseAdmin$9(979): Started disable of TestTable.testRowMutation 2015-04-04 10:48:06,123 INFO [B.defaultRpcServer.handler=3,queue=0,port=51327] master.HMaster(1659): Client=apurtell/null disable TestTable.testRowMutation 2015-04-04 10:48:06,125 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testRowMutation/write-master:513270000000001 2015-04-04 10:48:06,131 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(129): Attempting to disable table TestTable.testRowMutation 2015-04-04 10:48:06,131 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testRowMutation state from DISABLING to DISABLING 2015-04-04 10:48:06,132 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:48:06,133 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. (offlining), current state: {7cd74fdda353b2622f22dc40ab9c09f2 state=OPEN, ts=1428169686063, server=localhost,51330,1428169644428} 2015-04-04 10:48:06,133 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Creating unassigned node 7cd74fdda353b2622f22dc40ab9c09f2 in a CLOSING state 2015-04-04 10:48:06,134 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:06,134 INFO [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {7cd74fdda353b2622f22dc40ab9c09f2 state=OPEN, ts=1428169686063, server=localhost,51330,1428169644428} to {7cd74fdda353b2622f22dc40ab9c09f2 state=PENDING_CLOSE, ts=1428169686134, server=localhost,51330,1428169644428} 2015-04-04 10:48:06,134 INFO [PriorityRpcServer.handler=4,queue=0,port=51330] regionserver.RSRpcServices(1018): Close 7cd74fdda353b2622f22dc40ab9c09f2, moving to null 2015-04-04 10:48:06,135 DEBUG [RS_CLOSE_REGION-localhost:51330-2] handler.CloseRegionHandler(119): Processing close of TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. 2015-04-04 10:48:06,135 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,51330,1428169644428 for region TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. 2015-04-04 10:48:06,136 DEBUG [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegion(1350): Closing TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2.: disabling compactions & flushes 2015-04-04 10:48:06,136 DEBUG [Thread-485] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in TestTable.testRowMutation 2015-04-04 10:48:06,136 DEBUG [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegion(1377): Updates disabled for region TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. 2015-04-04 10:48:06,136 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegion(2084): Started memstore flush for TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2., current region memstore size 960 B, and 3/3 column families' memstores are being flushed. 2015-04-04 10:48:06,147 INFO [IPC Server handler 0 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741874_1050{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 0 2015-04-04 10:48:06,148 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=6, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/.tmp/85c47b9a2e7047b1ae42c7378ba6fda9 2015-04-04 10:48:06,164 INFO [IPC Server handler 2 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741875_1051{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 4726 2015-04-04 10:48:06,247 DEBUG [Thread-485] client.HBaseAdmin(1019): Sleeping= 200ms, waiting for all regions to be disabled in TestTable.testRowMutation 2015-04-04 10:48:06,459 DEBUG [Thread-485] client.HBaseAdmin(1019): Sleeping= 300ms, waiting for all regions to be disabled in TestTable.testRowMutation 2015-04-04 10:48:06,566 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=6, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/.tmp/03aecb507d544b45aca288be40d963ee 2015-04-04 10:48:06,590 INFO [IPC Server handler 4 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741876_1052{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 4726 2015-04-04 10:48:06,765 DEBUG [Thread-485] client.HBaseAdmin(1019): Sleeping= 500ms, waiting for all regions to be disabled in TestTable.testRowMutation 2015-04-04 10:48:06,996 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=6, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/.tmp/fda9559437fb4d5996d0adfd995a573b 2015-04-04 10:48:07,008 DEBUG [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/.tmp/85c47b9a2e7047b1ae42c7378ba6fda9 as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/a/85c47b9a2e7047b1ae42c7378ba6fda9 2015-04-04 10:48:07,017 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/a/85c47b9a2e7047b1ae42c7378ba6fda9, entries=1, sequenceid=6, filesize=4.6 K 2015-04-04 10:48:07,019 DEBUG [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/.tmp/03aecb507d544b45aca288be40d963ee as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/b/03aecb507d544b45aca288be40d963ee 2015-04-04 10:48:07,027 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/b/03aecb507d544b45aca288be40d963ee, entries=1, sequenceid=6, filesize=4.6 K 2015-04-04 10:48:07,029 DEBUG [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/.tmp/fda9559437fb4d5996d0adfd995a573b as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/c/fda9559437fb4d5996d0adfd995a573b 2015-04-04 10:48:07,037 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/c/fda9559437fb4d5996d0adfd995a573b, entries=1, sequenceid=6, filesize=4.6 K 2015-04-04 10:48:07,039 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegion(2363): Finished memstore flush of ~960 B/960, currentsize=0 B/0 for region TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. in 903ms, sequenceid=6, compaction requested=false 2015-04-04 10:48:07,041 INFO [StoreCloserThread-TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2.-1] regionserver.HStore(879): Closed a 2015-04-04 10:48:07,042 INFO [StoreCloserThread-TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2.-1] regionserver.HStore(879): Closed b 2015-04-04 10:48:07,048 INFO [StoreCloserThread-TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2.-1] regionserver.HStore(879): Closed c 2015-04-04 10:48:07,054 DEBUG [RS_CLOSE_REGION-localhost:51330-2] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/recovered.edits/9.seqid to file, newSeqId=9, maxSeqId=2 2015-04-04 10:48:07,055 DEBUG [RS_CLOSE_REGION-localhost:51330-2] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:48:07,055 DEBUG [RS_CLOSE_REGION-localhost:51330-2] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:48:07,055 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegion(1471): Closed TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. 2015-04-04 10:48:07,056 DEBUG [RS_CLOSE_REGION-localhost:51330-2] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning 7cd74fdda353b2622f22dc40ab9c09f2 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:48:07,065 DEBUG [RS_CLOSE_REGION-localhost:51330-2] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node 7cd74fdda353b2622f22dc40ab9c09f2 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:48:07,065 DEBUG [RS_CLOSE_REGION-localhost:51330-2] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. on localhost,51330,1428169644428 2015-04-04 10:48:07,065 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/7cd74fdda353b2622f22dc40ab9c09f2 2015-04-04 10:48:07,066 DEBUG [RS_CLOSE_REGION-localhost:51330-2] handler.CloseRegionHandler(164): Closed TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. 2015-04-04 10:48:07,066 DEBUG [AM.ZK.Worker-pool3-t8] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,51330,1428169644428, region=7cd74fdda353b2622f22dc40ab9c09f2, current_state={7cd74fdda353b2622f22dc40ab9c09f2 state=PENDING_CLOSE, ts=1428169686134, server=localhost,51330,1428169644428} 2015-04-04 10:48:07,067 DEBUG [AM.ZK.Worker-pool3-t8] handler.ClosedRegionHandler(93): Handling CLOSED event for 7cd74fdda353b2622f22dc40ab9c09f2 2015-04-04 10:48:07,067 DEBUG [AM.ZK.Worker-pool3-t8] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. 2015-04-04 10:48:07,068 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/7cd74fdda353b2622f22dc40ab9c09f2 2015-04-04 10:48:07,068 DEBUG [AM.ZK.Worker-pool3-t8] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node 7cd74fdda353b2622f22dc40ab9c09f2 in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:48:07,069 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:07,069 INFO [AM.ZK.Worker-pool3-t8] master.RegionStates(1112): Transition {7cd74fdda353b2622f22dc40ab9c09f2 state=PENDING_CLOSE, ts=1428169686134, server=localhost,51330,1428169644428} to {7cd74fdda353b2622f22dc40ab9c09f2 state=OFFLINE, ts=1428169687069, server=localhost,51330,1428169644428} 2015-04-04 10:48:07,069 INFO [AM.ZK.Worker-pool3-t8] master.RegionStates(600): Offlined 7cd74fdda353b2622f22dc40ab9c09f2 from localhost,51330,1428169644428 2015-04-04 10:48:07,092 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testRowMutation state from DISABLING to DISABLED 2015-04-04 10:48:07,094 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(191): Disabled table, TestTable.testRowMutation, is done=true 2015-04-04 10:48:07,095 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testRowMutation/write-master:513270000000001 2015-04-04 10:48:07,277 INFO [Thread-485] client.HBaseAdmin(1034): Disabled TestTable.testRowMutation 2015-04-04 10:48:07,278 INFO [B.defaultRpcServer.handler=1,queue=0,port=51327] master.HMaster(1564): Client=apurtell/null delete TestTable.testRowMutation 2015-04-04 10:48:07,289 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testRowMutation/write-master:513270000000002 2015-04-04 10:48:07,294 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table TestTable.testRowMutation 2015-04-04 10:48:07,299 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:48:07,301 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => 7cd74fdda353b2622f22dc40ab9c09f2, NAME => 'TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:48:07,304 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(186): Archiving region TestTable.testRowMutation,,1428169685962.7cd74fdda353b2622f22dc40ab9c09f2. from FS 2015-04-04 10:48:07,304 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2 2015-04-04 10:48:07,309 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/recovered.edits] 2015-04-04 10:48:07,318 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/a/85c47b9a2e7047b1ae42c7378ba6fda9, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/a/85c47b9a2e7047b1ae42c7378ba6fda9 2015-04-04 10:48:07,324 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/b/03aecb507d544b45aca288be40d963ee, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/b/03aecb507d544b45aca288be40d963ee 2015-04-04 10:48:07,331 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/c/fda9559437fb4d5996d0adfd995a573b, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/c/fda9559437fb4d5996d0adfd995a573b 2015-04-04 10:48:07,338 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/recovered.edits/9.seqid, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2/recovered.edits/9.seqid 2015-04-04 10:48:07,338 INFO [IPC Server handler 5 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741873_1049 127.0.0.1:51320 2015-04-04 10:48:07,339 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRowMutation/7cd74fdda353b2622f22dc40ab9c09f2 2015-04-04 10:48:07,339 INFO [IPC Server handler 0 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741872_1048 127.0.0.1:51320 2015-04-04 10:48:07,340 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(196): Table 'TestTable.testRowMutation' archived! 2015-04-04 10:48:07,340 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(116): Removing 'TestTable.testRowMutation' descriptor. 2015-04-04 10:48:07,340 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(122): Removing 'TestTable.testRowMutation' from region states. 2015-04-04 10:48:07,341 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(126): Marking 'TestTable.testRowMutation' as deleted. 2015-04-04 10:48:07,346 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testRowMutation/write-master:513270000000002 2015-04-04 10:48:07,412 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51327] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = TestTable.testRowMutation org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRowMutation at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:48:07,413 INFO [Thread-485] client.HBaseAdmin(739): Deleted TestTable.testRowMutation 2015-04-04 10:48:07,443 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testRowMutation Thread=529 (was 503) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t129 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t132 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t99 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51416 [Waiting for operation #29] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t96 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51459 [Waiting for operation #3] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x34c6ce8e-shared--pool9-t14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t103 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t122 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t127 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t102 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t100 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t121 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t128 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t94 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51417 [Waiting for operation #27] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t98 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t131 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t125 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t101 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t124 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t97 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t95 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t126 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t130 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t123 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t105 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t104 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=438 (was 439), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=232 (was 217) - SystemLoadAverage LEAK? -, ProcessCount=232 (was 232), AvailableMemoryMB=16 (was 17), ConnectionCount=1 (was 1) 2015-04-04 10:48:07,444 WARN [main] hbase.ResourceChecker(134): Thread=529 is superior to 500 2015-04-04 10:48:07,488 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testRegionObserver Thread=529, OpenFileDescriptor=438, MaxFileDescriptor=10240, SystemLoadAverage=232, ProcessCount=232, AvailableMemoryMB=15, ConnectionCount=1 2015-04-04 10:48:07,488 WARN [main] hbase.ResourceChecker(134): Thread=529 is superior to 500 2015-04-04 10:48:07,490 INFO [B.defaultRpcServer.handler=0,queue=0,port=51327] master.HMaster(1318): Client=apurtell/null create 'TestTable.testRegionObserver', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:48:07,494 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testRegionObserver/write-master:513270000000000 2015-04-04 10:48:07,499 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(188): Create table TestTable.testRegionObserver 2015-04-04 10:48:07,509 INFO [IPC Server handler 3 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741877_1053{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 0 2015-04-04 10:48:07,511 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRegionObserver/.tabledesc/.tableinfo.0000000001 2015-04-04 10:48:07,513 INFO [RegionOpenAndInitThread-TestTable.testRegionObserver-1] regionserver.HRegion(5885): creating HRegion TestTable.testRegionObserver HTD == 'TestTable.testRegionObserver', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp Table name == TestTable.testRegionObserver 2015-04-04 10:48:07,521 INFO [IPC Server handler 3 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741878_1054{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 0 2015-04-04 10:48:07,522 DEBUG [RegionOpenAndInitThread-TestTable.testRegionObserver-1] regionserver.HRegion(721): Instantiated TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. 2015-04-04 10:48:07,522 DEBUG [RegionOpenAndInitThread-TestTable.testRegionObserver-1] regionserver.HRegion(1350): Closing TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b.: disabling compactions & flushes 2015-04-04 10:48:07,522 DEBUG [RegionOpenAndInitThread-TestTable.testRegionObserver-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. 2015-04-04 10:48:07,522 INFO [RegionOpenAndInitThread-TestTable.testRegionObserver-1] regionserver.HRegion(1471): Closed TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. 2015-04-04 10:48:07,526 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:48:07,527 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,51330,1428169644428 2015-04-04 10:48:07,527 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKAssign(174): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Async create of unassigned node aa2470c5a009675eb0e9144de8fdc19b with OFFLINE state 2015-04-04 10:48:07,528 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:07,528 DEBUG [main-EventThread] master.OfflineCallback(69): rs={aa2470c5a009675eb0e9144de8fdc19b state=OFFLINE, ts=1428169687527, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:48:07,528 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={aa2470c5a009675eb0e9144de8fdc19b state=OFFLINE, ts=1428169687527, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:48:07,534 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1658): localhost,51330,1428169644428 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:48:07,534 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.RegionStates(1112): Transition {aa2470c5a009675eb0e9144de8fdc19b state=OFFLINE, ts=1428169687527, server=null} to {aa2470c5a009675eb0e9144de8fdc19b state=PENDING_OPEN, ts=1428169687534, server=localhost,51330,1428169644428} 2015-04-04 10:48:07,535 INFO [PriorityRpcServer.handler=7,queue=1,port=51330] regionserver.RSRpcServices(1390): Open TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. 2015-04-04 10:48:07,539 DEBUG [RS_OPEN_REGION-localhost:51330-2] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning aa2470c5a009675eb0e9144de8fdc19b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:48:07,539 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1788): Bulk assigning done for localhost,51330,1428169644428 2015-04-04 10:48:07,539 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testRegionObserver state from ENABLING to ENABLED 2015-04-04 10:48:07,540 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/aa2470c5a009675eb0e9144de8fdc19b 2015-04-04 10:48:07,540 DEBUG [RS_OPEN_REGION-localhost:51330-2] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node aa2470c5a009675eb0e9144de8fdc19b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:48:07,541 DEBUG [RS_OPEN_REGION-localhost:51330-2] regionserver.HRegion(6078): Opening region: {ENCODED => aa2470c5a009675eb0e9144de8fdc19b, NAME => 'TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:48:07,541 DEBUG [AM.ZK.Worker-pool3-t13] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,51330,1428169644428, region=aa2470c5a009675eb0e9144de8fdc19b, current_state={aa2470c5a009675eb0e9144de8fdc19b state=PENDING_OPEN, ts=1428169687534, server=localhost,51330,1428169644428} 2015-04-04 10:48:07,541 INFO [AM.ZK.Worker-pool3-t13] master.RegionStates(1112): Transition {aa2470c5a009675eb0e9144de8fdc19b state=PENDING_OPEN, ts=1428169687534, server=localhost,51330,1428169644428} to {aa2470c5a009675eb0e9144de8fdc19b state=OPENING, ts=1428169687541, server=localhost,51330,1428169644428} 2015-04-04 10:48:07,541 DEBUG [RS_OPEN_REGION-localhost:51330-2] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:48:07,541 DEBUG [RS_OPEN_REGION-localhost:51330-2] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:48:07,542 INFO [RS_OPEN_REGION-localhost:51330-2] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:48:07,542 INFO [RS_OPEN_REGION-localhost:51330-2] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:48:07,542 DEBUG [RS_OPEN_REGION-localhost:51330-2] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testRegionObserver aa2470c5a009675eb0e9144de8fdc19b 2015-04-04 10:48:07,542 DEBUG [RS_OPEN_REGION-localhost:51330-2] regionserver.HRegion(721): Instantiated TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. 2015-04-04 10:48:07,545 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testRegionObserver/write-master:513270000000000 2015-04-04 10:48:07,545 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(218): Table, TestTable.testRegionObserver, creation successful 2015-04-04 10:48:07,546 INFO [StoreOpener-aa2470c5a009675eb0e9144de8fdc19b-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:07,547 INFO [StoreOpener-aa2470c5a009675eb0e9144de8fdc19b-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:07,548 DEBUG [StoreOpener-aa2470c5a009675eb0e9144de8fdc19b-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/a 2015-04-04 10:48:07,550 INFO [StoreOpener-aa2470c5a009675eb0e9144de8fdc19b-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:07,551 INFO [StoreOpener-aa2470c5a009675eb0e9144de8fdc19b-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:07,552 DEBUG [StoreOpener-aa2470c5a009675eb0e9144de8fdc19b-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/b 2015-04-04 10:48:07,555 INFO [StoreOpener-aa2470c5a009675eb0e9144de8fdc19b-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:07,555 INFO [StoreOpener-aa2470c5a009675eb0e9144de8fdc19b-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:07,556 DEBUG [StoreOpener-aa2470c5a009675eb0e9144de8fdc19b-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/c 2015-04-04 10:48:07,558 DEBUG [RS_OPEN_REGION-localhost:51330-2] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b 2015-04-04 10:48:07,560 DEBUG [RS_OPEN_REGION-localhost:51330-2] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:48:07,565 DEBUG [RS_OPEN_REGION-localhost:51330-2] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:48:07,565 INFO [RS_OPEN_REGION-localhost:51330-2] regionserver.HRegion(847): Onlined aa2470c5a009675eb0e9144de8fdc19b; next sequenceid=2 2015-04-04 10:48:07,566 DEBUG [RS_OPEN_REGION-localhost:51330-2] zookeeper.ZKAssign(643): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Attempting to retransition opening state of node aa2470c5a009675eb0e9144de8fdc19b 2015-04-04 10:48:07,568 INFO [PostOpenDeployTasks:aa2470c5a009675eb0e9144de8fdc19b] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. 2015-04-04 10:48:07,571 INFO [PostOpenDeployTasks:aa2470c5a009675eb0e9144de8fdc19b] hbase.MetaTableAccessor(1350): Updated row TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. with server=localhost,51330,1428169644428 2015-04-04 10:48:07,571 DEBUG [PostOpenDeployTasks:aa2470c5a009675eb0e9144de8fdc19b] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. 2015-04-04 10:48:07,572 DEBUG [RS_OPEN_REGION-localhost:51330-2] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning aa2470c5a009675eb0e9144de8fdc19b from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:48:07,574 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/aa2470c5a009675eb0e9144de8fdc19b 2015-04-04 10:48:07,574 DEBUG [RS_OPEN_REGION-localhost:51330-2] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node aa2470c5a009675eb0e9144de8fdc19b from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:48:07,574 DEBUG [RS_OPEN_REGION-localhost:51330-2] coordination.ZkOpenRegionCoordination(91): Transitioned aa2470c5a009675eb0e9144de8fdc19b to OPENED in zk on localhost,51330,1428169644428 2015-04-04 10:48:07,574 DEBUG [RS_OPEN_REGION-localhost:51330-2] handler.OpenRegionHandler(167): Opened TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. on localhost,51330,1428169644428 2015-04-04 10:48:07,575 DEBUG [AM.ZK.Worker-pool3-t11] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,51330,1428169644428, region=aa2470c5a009675eb0e9144de8fdc19b, current_state={aa2470c5a009675eb0e9144de8fdc19b state=OPENING, ts=1428169687541, server=localhost,51330,1428169644428} 2015-04-04 10:48:07,575 INFO [AM.ZK.Worker-pool3-t11] master.RegionStates(1112): Transition {aa2470c5a009675eb0e9144de8fdc19b state=OPENING, ts=1428169687541, server=localhost,51330,1428169644428} to {aa2470c5a009675eb0e9144de8fdc19b state=OPEN, ts=1428169687575, server=localhost,51330,1428169644428} 2015-04-04 10:48:07,575 DEBUG [AM.ZK.Worker-pool3-t11] coordination.ZkOpenRegionCoordination(350): Handling OPENED of aa2470c5a009675eb0e9144de8fdc19b from localhost,51327,1428169644129; deleting unassigned node 2015-04-04 10:48:07,576 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/aa2470c5a009675eb0e9144de8fdc19b 2015-04-04 10:48:07,576 DEBUG [AM.ZK.Worker-pool3-t11] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node aa2470c5a009675eb0e9144de8fdc19b in expected state RS_ZK_REGION_OPENED 2015-04-04 10:48:07,576 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:07,576 DEBUG [AM.ZK.Worker-pool3-t11] master.AssignmentManager$4(1337): Znode TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. deleted, state: {aa2470c5a009675eb0e9144de8fdc19b state=OPEN, ts=1428169687575, server=localhost,51330,1428169644428} 2015-04-04 10:48:07,576 DEBUG [AM.ZK.Worker-pool3-t11] master.RegionStates(460): Onlined aa2470c5a009675eb0e9144de8fdc19b on localhost,51330,1428169644428 {ENCODED => aa2470c5a009675eb0e9144de8fdc19b, NAME => 'TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:48:07,616 INFO [Thread-509] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:48:07,621 INFO [Thread-509] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:48:07,634 INFO [Thread-509] client.HBaseAdmin$9(979): Started disable of TestTable.testRegionObserver 2015-04-04 10:48:07,635 INFO [B.defaultRpcServer.handler=2,queue=0,port=51327] master.HMaster(1659): Client=apurtell/null disable TestTable.testRegionObserver 2015-04-04 10:48:07,637 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testRegionObserver/write-master:513270000000001 2015-04-04 10:48:07,642 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(129): Attempting to disable table TestTable.testRegionObserver 2015-04-04 10:48:07,643 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testRegionObserver state from DISABLING to DISABLING 2015-04-04 10:48:07,644 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:48:07,644 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. (offlining), current state: {aa2470c5a009675eb0e9144de8fdc19b state=OPEN, ts=1428169687576, server=localhost,51330,1428169644428} 2015-04-04 10:48:07,645 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Creating unassigned node aa2470c5a009675eb0e9144de8fdc19b in a CLOSING state 2015-04-04 10:48:07,645 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:07,646 INFO [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {aa2470c5a009675eb0e9144de8fdc19b state=OPEN, ts=1428169687576, server=localhost,51330,1428169644428} to {aa2470c5a009675eb0e9144de8fdc19b state=PENDING_CLOSE, ts=1428169687646, server=localhost,51330,1428169644428} 2015-04-04 10:48:07,646 INFO [PriorityRpcServer.handler=4,queue=0,port=51330] regionserver.RSRpcServices(1018): Close aa2470c5a009675eb0e9144de8fdc19b, moving to null 2015-04-04 10:48:07,647 DEBUG [RS_CLOSE_REGION-localhost:51330-0] handler.CloseRegionHandler(119): Processing close of TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. 2015-04-04 10:48:07,647 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,51330,1428169644428 for region TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. 2015-04-04 10:48:07,648 DEBUG [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(1350): Closing TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b.: disabling compactions & flushes 2015-04-04 10:48:07,648 DEBUG [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(1377): Updates disabled for region TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. 2015-04-04 10:48:07,648 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(2084): Started memstore flush for TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b., current region memstore size 960 B, and 3/3 column families' memstores are being flushed. 2015-04-04 10:48:07,648 DEBUG [Thread-509] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in TestTable.testRegionObserver 2015-04-04 10:48:07,658 INFO [IPC Server handler 3 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741879_1055{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 4726 2015-04-04 10:48:07,759 DEBUG [Thread-509] client.HBaseAdmin(1019): Sleeping= 200ms, waiting for all regions to be disabled in TestTable.testRegionObserver 2015-04-04 10:48:07,969 DEBUG [Thread-509] client.HBaseAdmin(1019): Sleeping= 300ms, waiting for all regions to be disabled in TestTable.testRegionObserver 2015-04-04 10:48:08,061 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/.tmp/72b70f80a7764f83abbd344259d252c8 2015-04-04 10:48:08,078 INFO [IPC Server handler 0 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741880_1056{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 0 2015-04-04 10:48:08,079 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/.tmp/fb0fd95c28a44526acc75c8e32b85a5e 2015-04-04 10:48:08,097 INFO [IPC Server handler 8 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741881_1057{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 0 2015-04-04 10:48:08,098 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=320, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/.tmp/ccf662c0a0bb426ba07ad5eb618a689f 2015-04-04 10:48:08,106 DEBUG [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/.tmp/72b70f80a7764f83abbd344259d252c8 as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/a/72b70f80a7764f83abbd344259d252c8 2015-04-04 10:48:08,114 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/a/72b70f80a7764f83abbd344259d252c8, entries=1, sequenceid=8, filesize=4.6 K 2015-04-04 10:48:08,116 DEBUG [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/.tmp/fb0fd95c28a44526acc75c8e32b85a5e as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/b/fb0fd95c28a44526acc75c8e32b85a5e 2015-04-04 10:48:08,123 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/b/fb0fd95c28a44526acc75c8e32b85a5e, entries=1, sequenceid=8, filesize=4.6 K 2015-04-04 10:48:08,125 DEBUG [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/.tmp/ccf662c0a0bb426ba07ad5eb618a689f as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/c/ccf662c0a0bb426ba07ad5eb618a689f 2015-04-04 10:48:08,133 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/c/ccf662c0a0bb426ba07ad5eb618a689f, entries=1, sequenceid=8, filesize=4.6 K 2015-04-04 10:48:08,134 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(2363): Finished memstore flush of ~960 B/960, currentsize=0 B/0 for region TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. in 486ms, sequenceid=8, compaction requested=false 2015-04-04 10:48:08,137 INFO [StoreCloserThread-TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b.-1] regionserver.HStore(879): Closed a 2015-04-04 10:48:08,138 INFO [StoreCloserThread-TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b.-1] regionserver.HStore(879): Closed b 2015-04-04 10:48:08,140 INFO [StoreCloserThread-TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b.-1] regionserver.HStore(879): Closed c 2015-04-04 10:48:08,148 DEBUG [RS_CLOSE_REGION-localhost:51330-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/recovered.edits/11.seqid to file, newSeqId=11, maxSeqId=2 2015-04-04 10:48:08,149 DEBUG [RS_CLOSE_REGION-localhost:51330-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:48:08,149 DEBUG [RS_CLOSE_REGION-localhost:51330-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:48:08,149 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(1471): Closed TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. 2015-04-04 10:48:08,149 DEBUG [RS_CLOSE_REGION-localhost:51330-0] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning aa2470c5a009675eb0e9144de8fdc19b from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:48:08,151 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/aa2470c5a009675eb0e9144de8fdc19b 2015-04-04 10:48:08,151 DEBUG [RS_CLOSE_REGION-localhost:51330-0] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node aa2470c5a009675eb0e9144de8fdc19b from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:48:08,151 DEBUG [RS_CLOSE_REGION-localhost:51330-0] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. on localhost,51330,1428169644428 2015-04-04 10:48:08,151 DEBUG [RS_CLOSE_REGION-localhost:51330-0] handler.CloseRegionHandler(164): Closed TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. 2015-04-04 10:48:08,151 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,51330,1428169644428, region=aa2470c5a009675eb0e9144de8fdc19b, current_state={aa2470c5a009675eb0e9144de8fdc19b state=PENDING_CLOSE, ts=1428169687646, server=localhost,51330,1428169644428} 2015-04-04 10:48:08,151 DEBUG [AM.ZK.Worker-pool3-t14] handler.ClosedRegionHandler(93): Handling CLOSED event for aa2470c5a009675eb0e9144de8fdc19b 2015-04-04 10:48:08,151 DEBUG [AM.ZK.Worker-pool3-t14] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. 2015-04-04 10:48:08,153 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/aa2470c5a009675eb0e9144de8fdc19b 2015-04-04 10:48:08,153 DEBUG [AM.ZK.Worker-pool3-t14] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node aa2470c5a009675eb0e9144de8fdc19b in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:48:08,153 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:08,153 INFO [AM.ZK.Worker-pool3-t14] master.RegionStates(1112): Transition {aa2470c5a009675eb0e9144de8fdc19b state=PENDING_CLOSE, ts=1428169687646, server=localhost,51330,1428169644428} to {aa2470c5a009675eb0e9144de8fdc19b state=OFFLINE, ts=1428169688153, server=localhost,51330,1428169644428} 2015-04-04 10:48:08,153 INFO [AM.ZK.Worker-pool3-t14] master.RegionStates(600): Offlined aa2470c5a009675eb0e9144de8fdc19b from localhost,51330,1428169644428 2015-04-04 10:48:08,170 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testRegionObserver state from DISABLING to DISABLED 2015-04-04 10:48:08,172 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(191): Disabled table, TestTable.testRegionObserver, is done=true 2015-04-04 10:48:08,173 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testRegionObserver/write-master:513270000000001 2015-04-04 10:48:08,280 INFO [Thread-509] client.HBaseAdmin(1034): Disabled TestTable.testRegionObserver 2015-04-04 10:48:08,282 INFO [B.defaultRpcServer.handler=0,queue=0,port=51327] master.HMaster(1564): Client=apurtell/null delete TestTable.testRegionObserver 2015-04-04 10:48:08,292 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testRegionObserver/write-master:513270000000002 2015-04-04 10:48:08,298 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table TestTable.testRegionObserver 2015-04-04 10:48:08,302 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:48:08,305 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => aa2470c5a009675eb0e9144de8fdc19b, NAME => 'TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:48:08,307 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(186): Archiving region TestTable.testRegionObserver,,1428169687490.aa2470c5a009675eb0e9144de8fdc19b. from FS 2015-04-04 10:48:08,308 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b 2015-04-04 10:48:08,313 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/recovered.edits] 2015-04-04 10:48:08,320 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/a/72b70f80a7764f83abbd344259d252c8, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/a/72b70f80a7764f83abbd344259d252c8 2015-04-04 10:48:08,326 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/b/fb0fd95c28a44526acc75c8e32b85a5e, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/b/fb0fd95c28a44526acc75c8e32b85a5e 2015-04-04 10:48:08,332 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/c/ccf662c0a0bb426ba07ad5eb618a689f, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/c/ccf662c0a0bb426ba07ad5eb618a689f 2015-04-04 10:48:08,338 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/recovered.edits/11.seqid, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b/recovered.edits/11.seqid 2015-04-04 10:48:08,339 INFO [IPC Server handler 7 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741878_1054 127.0.0.1:51320 2015-04-04 10:48:08,340 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testRegionObserver/aa2470c5a009675eb0e9144de8fdc19b 2015-04-04 10:48:08,341 INFO [IPC Server handler 1 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741877_1053 127.0.0.1:51320 2015-04-04 10:48:08,341 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(196): Table 'TestTable.testRegionObserver' archived! 2015-04-04 10:48:08,341 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(116): Removing 'TestTable.testRegionObserver' descriptor. 2015-04-04 10:48:08,342 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(122): Removing 'TestTable.testRegionObserver' from region states. 2015-04-04 10:48:08,342 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(126): Marking 'TestTable.testRegionObserver' as deleted. 2015-04-04 10:48:08,347 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testRegionObserver/write-master:513270000000002 2015-04-04 10:48:08,412 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51327] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = TestTable.testRegionObserver org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testRegionObserver at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:48:08,413 INFO [Thread-509] client.HBaseAdmin(739): Deleted TestTable.testRegionObserver 2015-04-04 10:48:08,444 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testRegionObserver Thread=551 (was 529) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t113 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t107 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51416 [Waiting for operation #33] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51459 [Waiting for operation #7] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t135 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t136 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t115 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t140 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t144 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t133 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t111 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t143 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51417 [Waiting for operation #33] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t110 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t109 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t112 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t134 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t114 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x34c6ce8e-shared--pool9-t15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t108 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t106 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t116 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t137 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t139 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t138 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t141 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t142 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=435 (was 438), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=232 (was 232), ProcessCount=232 (was 232), AvailableMemoryMB=16 (was 15) - AvailableMemoryMB LEAK? -, ConnectionCount=1 (was 1) 2015-04-04 10:48:08,444 WARN [main] hbase.ResourceChecker(134): Thread=551 is superior to 500 2015-04-04 10:48:08,474 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testIncrementHook Thread=551, OpenFileDescriptor=435, MaxFileDescriptor=10240, SystemLoadAverage=232, ProcessCount=232, AvailableMemoryMB=16, ConnectionCount=1 2015-04-04 10:48:08,474 WARN [main] hbase.ResourceChecker(134): Thread=551 is superior to 500 2015-04-04 10:48:08,476 INFO [B.defaultRpcServer.handler=4,queue=0,port=51327] master.HMaster(1318): Client=apurtell/null create 'TestTable.testIncrementHook', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:48:08,489 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testIncrementHook/write-master:513270000000000 2015-04-04 10:48:08,497 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(188): Create table TestTable.testIncrementHook 2015-04-04 10:48:08,506 INFO [IPC Server handler 7 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741882_1058{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 0 2015-04-04 10:48:08,508 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testIncrementHook/.tabledesc/.tableinfo.0000000001 2015-04-04 10:48:08,510 INFO [RegionOpenAndInitThread-TestTable.testIncrementHook-1] regionserver.HRegion(5885): creating HRegion TestTable.testIncrementHook HTD == 'TestTable.testIncrementHook', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp Table name == TestTable.testIncrementHook 2015-04-04 10:48:08,518 INFO [IPC Server handler 7 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741883_1059{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 0 2015-04-04 10:48:08,519 DEBUG [RegionOpenAndInitThread-TestTable.testIncrementHook-1] regionserver.HRegion(721): Instantiated TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. 2015-04-04 10:48:08,519 DEBUG [RegionOpenAndInitThread-TestTable.testIncrementHook-1] regionserver.HRegion(1350): Closing TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665.: disabling compactions & flushes 2015-04-04 10:48:08,519 DEBUG [RegionOpenAndInitThread-TestTable.testIncrementHook-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. 2015-04-04 10:48:08,519 INFO [RegionOpenAndInitThread-TestTable.testIncrementHook-1] regionserver.HRegion(1471): Closed TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. 2015-04-04 10:48:08,523 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:48:08,524 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,51330,1428169644428 2015-04-04 10:48:08,524 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKAssign(174): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Async create of unassigned node 32117afaa6ee9f522aba3ba6ee438665 with OFFLINE state 2015-04-04 10:48:08,525 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:08,525 DEBUG [main-EventThread] master.OfflineCallback(69): rs={32117afaa6ee9f522aba3ba6ee438665 state=OFFLINE, ts=1428169688524, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:48:08,526 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={32117afaa6ee9f522aba3ba6ee438665 state=OFFLINE, ts=1428169688524, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:48:08,531 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1658): localhost,51330,1428169644428 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:48:08,531 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.RegionStates(1112): Transition {32117afaa6ee9f522aba3ba6ee438665 state=OFFLINE, ts=1428169688524, server=null} to {32117afaa6ee9f522aba3ba6ee438665 state=PENDING_OPEN, ts=1428169688531, server=localhost,51330,1428169644428} 2015-04-04 10:48:08,532 INFO [PriorityRpcServer.handler=0,queue=0,port=51330] regionserver.RSRpcServices(1390): Open TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. 2015-04-04 10:48:08,538 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning 32117afaa6ee9f522aba3ba6ee438665 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:48:08,538 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1788): Bulk assigning done for localhost,51330,1428169644428 2015-04-04 10:48:08,538 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testIncrementHook state from ENABLING to ENABLED 2015-04-04 10:48:08,540 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/32117afaa6ee9f522aba3ba6ee438665 2015-04-04 10:48:08,540 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node 32117afaa6ee9f522aba3ba6ee438665 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:48:08,540 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(6078): Opening region: {ENCODED => 32117afaa6ee9f522aba3ba6ee438665, NAME => 'TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:48:08,541 DEBUG [AM.ZK.Worker-pool3-t18] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,51330,1428169644428, region=32117afaa6ee9f522aba3ba6ee438665, current_state={32117afaa6ee9f522aba3ba6ee438665 state=PENDING_OPEN, ts=1428169688531, server=localhost,51330,1428169644428} 2015-04-04 10:48:08,541 DEBUG [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:48:08,541 INFO [AM.ZK.Worker-pool3-t18] master.RegionStates(1112): Transition {32117afaa6ee9f522aba3ba6ee438665 state=PENDING_OPEN, ts=1428169688531, server=localhost,51330,1428169644428} to {32117afaa6ee9f522aba3ba6ee438665 state=OPENING, ts=1428169688541, server=localhost,51330,1428169644428} 2015-04-04 10:48:08,541 DEBUG [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:48:08,824 INFO [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:48:08,825 INFO [RS_OPEN_REGION-localhost:51330-0] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:48:08,826 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testIncrementHook 32117afaa6ee9f522aba3ba6ee438665 2015-04-04 10:48:08,826 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(721): Instantiated TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. 2015-04-04 10:48:08,830 INFO [StoreOpener-32117afaa6ee9f522aba3ba6ee438665-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:08,831 INFO [StoreOpener-32117afaa6ee9f522aba3ba6ee438665-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:08,832 DEBUG [StoreOpener-32117afaa6ee9f522aba3ba6ee438665-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/a 2015-04-04 10:48:08,834 INFO [StoreOpener-32117afaa6ee9f522aba3ba6ee438665-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:08,834 INFO [StoreOpener-32117afaa6ee9f522aba3ba6ee438665-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:08,835 DEBUG [StoreOpener-32117afaa6ee9f522aba3ba6ee438665-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/b 2015-04-04 10:48:08,837 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testIncrementHook/write-master:513270000000000 2015-04-04 10:48:08,837 INFO [StoreOpener-32117afaa6ee9f522aba3ba6ee438665-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:08,837 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(218): Table, TestTable.testIncrementHook, creation successful 2015-04-04 10:48:08,838 INFO [StoreOpener-32117afaa6ee9f522aba3ba6ee438665-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:08,839 DEBUG [StoreOpener-32117afaa6ee9f522aba3ba6ee438665-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/c 2015-04-04 10:48:08,839 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665 2015-04-04 10:48:08,842 DEBUG [RS_OPEN_REGION-localhost:51330-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:48:08,846 DEBUG [RS_OPEN_REGION-localhost:51330-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:48:08,846 INFO [RS_OPEN_REGION-localhost:51330-0] regionserver.HRegion(847): Onlined 32117afaa6ee9f522aba3ba6ee438665; next sequenceid=2 2015-04-04 10:48:08,847 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(643): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Attempting to retransition opening state of node 32117afaa6ee9f522aba3ba6ee438665 2015-04-04 10:48:08,848 INFO [PostOpenDeployTasks:32117afaa6ee9f522aba3ba6ee438665] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. 2015-04-04 10:48:08,851 INFO [PostOpenDeployTasks:32117afaa6ee9f522aba3ba6ee438665] hbase.MetaTableAccessor(1350): Updated row TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. with server=localhost,51330,1428169644428 2015-04-04 10:48:08,851 DEBUG [PostOpenDeployTasks:32117afaa6ee9f522aba3ba6ee438665] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. 2015-04-04 10:48:08,852 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning 32117afaa6ee9f522aba3ba6ee438665 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:48:08,854 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/32117afaa6ee9f522aba3ba6ee438665 2015-04-04 10:48:08,854 DEBUG [RS_OPEN_REGION-localhost:51330-0] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node 32117afaa6ee9f522aba3ba6ee438665 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:48:08,854 DEBUG [RS_OPEN_REGION-localhost:51330-0] coordination.ZkOpenRegionCoordination(91): Transitioned 32117afaa6ee9f522aba3ba6ee438665 to OPENED in zk on localhost,51330,1428169644428 2015-04-04 10:48:08,854 DEBUG [RS_OPEN_REGION-localhost:51330-0] handler.OpenRegionHandler(167): Opened TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. on localhost,51330,1428169644428 2015-04-04 10:48:08,855 DEBUG [AM.ZK.Worker-pool3-t3] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,51330,1428169644428, region=32117afaa6ee9f522aba3ba6ee438665, current_state={32117afaa6ee9f522aba3ba6ee438665 state=OPENING, ts=1428169688541, server=localhost,51330,1428169644428} 2015-04-04 10:48:08,855 INFO [AM.ZK.Worker-pool3-t3] master.RegionStates(1112): Transition {32117afaa6ee9f522aba3ba6ee438665 state=OPENING, ts=1428169688541, server=localhost,51330,1428169644428} to {32117afaa6ee9f522aba3ba6ee438665 state=OPEN, ts=1428169688855, server=localhost,51330,1428169644428} 2015-04-04 10:48:08,855 DEBUG [AM.ZK.Worker-pool3-t3] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 32117afaa6ee9f522aba3ba6ee438665 from localhost,51327,1428169644129; deleting unassigned node 2015-04-04 10:48:08,856 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/32117afaa6ee9f522aba3ba6ee438665 2015-04-04 10:48:08,856 DEBUG [AM.ZK.Worker-pool3-t3] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node 32117afaa6ee9f522aba3ba6ee438665 in expected state RS_ZK_REGION_OPENED 2015-04-04 10:48:08,856 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:08,856 DEBUG [AM.ZK.Worker-pool3-t3] master.AssignmentManager$4(1337): Znode TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. deleted, state: {32117afaa6ee9f522aba3ba6ee438665 state=OPEN, ts=1428169688855, server=localhost,51330,1428169644428} 2015-04-04 10:48:08,857 DEBUG [AM.ZK.Worker-pool3-t3] master.RegionStates(460): Onlined 32117afaa6ee9f522aba3ba6ee438665 on localhost,51330,1428169644428 {ENCODED => 32117afaa6ee9f522aba3ba6ee438665, NAME => 'TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:48:09,046 INFO [Thread-532] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:48:09,052 INFO [Thread-532] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:48:09,072 INFO [Thread-532] client.HBaseAdmin$9(979): Started disable of TestTable.testIncrementHook 2015-04-04 10:48:09,073 INFO [B.defaultRpcServer.handler=2,queue=0,port=51327] master.HMaster(1659): Client=apurtell/null disable TestTable.testIncrementHook 2015-04-04 10:48:09,075 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testIncrementHook/write-master:513270000000001 2015-04-04 10:48:09,081 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(129): Attempting to disable table TestTable.testIncrementHook 2015-04-04 10:48:09,081 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testIncrementHook state from DISABLING to DISABLING 2015-04-04 10:48:09,082 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:48:09,083 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. (offlining), current state: {32117afaa6ee9f522aba3ba6ee438665 state=OPEN, ts=1428169688857, server=localhost,51330,1428169644428} 2015-04-04 10:48:09,083 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Creating unassigned node 32117afaa6ee9f522aba3ba6ee438665 in a CLOSING state 2015-04-04 10:48:09,084 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:09,084 INFO [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {32117afaa6ee9f522aba3ba6ee438665 state=OPEN, ts=1428169688857, server=localhost,51330,1428169644428} to {32117afaa6ee9f522aba3ba6ee438665 state=PENDING_CLOSE, ts=1428169689084, server=localhost,51330,1428169644428} 2015-04-04 10:48:09,085 INFO [PriorityRpcServer.handler=2,queue=0,port=51330] regionserver.RSRpcServices(1018): Close 32117afaa6ee9f522aba3ba6ee438665, moving to null 2015-04-04 10:48:09,085 DEBUG [RS_CLOSE_REGION-localhost:51330-1] handler.CloseRegionHandler(119): Processing close of TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. 2015-04-04 10:48:09,085 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,51330,1428169644428 for region TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. 2015-04-04 10:48:09,086 DEBUG [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(1350): Closing TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665.: disabling compactions & flushes 2015-04-04 10:48:09,086 DEBUG [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. 2015-04-04 10:48:09,086 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(2084): Started memstore flush for TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665., current region memstore size 168 B, and 3/3 column families' memstores are being flushed. 2015-04-04 10:48:09,087 DEBUG [Thread-532] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in TestTable.testIncrementHook 2015-04-04 10:48:09,098 INFO [IPC Server handler 7 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741884_1060{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 0 2015-04-04 10:48:09,098 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=6, memsize=168, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/.tmp/2020a03eefba4920843651cf4e920257 2015-04-04 10:48:09,106 DEBUG [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/.tmp/2020a03eefba4920843651cf4e920257 as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/a/2020a03eefba4920843651cf4e920257 2015-04-04 10:48:09,114 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/a/2020a03eefba4920843651cf4e920257, entries=1, sequenceid=6, filesize=4.6 K 2015-04-04 10:48:09,115 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(2363): Finished memstore flush of ~168 B/168, currentsize=0 B/0 for region TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. in 29ms, sequenceid=6, compaction requested=false 2015-04-04 10:48:09,117 INFO [StoreCloserThread-TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665.-1] regionserver.HStore(879): Closed a 2015-04-04 10:48:09,117 INFO [StoreCloserThread-TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665.-1] regionserver.HStore(879): Closed b 2015-04-04 10:48:09,117 INFO [StoreCloserThread-TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665.-1] regionserver.HStore(879): Closed c 2015-04-04 10:48:09,125 DEBUG [RS_CLOSE_REGION-localhost:51330-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/recovered.edits/9.seqid to file, newSeqId=9, maxSeqId=2 2015-04-04 10:48:09,126 DEBUG [RS_CLOSE_REGION-localhost:51330-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:48:09,126 DEBUG [RS_CLOSE_REGION-localhost:51330-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:48:09,126 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(1471): Closed TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. 2015-04-04 10:48:09,126 DEBUG [RS_CLOSE_REGION-localhost:51330-1] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning 32117afaa6ee9f522aba3ba6ee438665 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:48:09,128 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/32117afaa6ee9f522aba3ba6ee438665 2015-04-04 10:48:09,128 DEBUG [RS_CLOSE_REGION-localhost:51330-1] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node 32117afaa6ee9f522aba3ba6ee438665 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:48:09,128 DEBUG [RS_CLOSE_REGION-localhost:51330-1] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. on localhost,51330,1428169644428 2015-04-04 10:48:09,128 DEBUG [RS_CLOSE_REGION-localhost:51330-1] handler.CloseRegionHandler(164): Closed TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. 2015-04-04 10:48:09,128 DEBUG [AM.ZK.Worker-pool3-t9] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,51330,1428169644428, region=32117afaa6ee9f522aba3ba6ee438665, current_state={32117afaa6ee9f522aba3ba6ee438665 state=PENDING_CLOSE, ts=1428169689084, server=localhost,51330,1428169644428} 2015-04-04 10:48:09,128 DEBUG [AM.ZK.Worker-pool3-t9] handler.ClosedRegionHandler(93): Handling CLOSED event for 32117afaa6ee9f522aba3ba6ee438665 2015-04-04 10:48:09,128 DEBUG [AM.ZK.Worker-pool3-t9] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. 2015-04-04 10:48:09,130 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/32117afaa6ee9f522aba3ba6ee438665 2015-04-04 10:48:09,130 DEBUG [AM.ZK.Worker-pool3-t9] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node 32117afaa6ee9f522aba3ba6ee438665 in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:48:09,130 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:09,130 INFO [AM.ZK.Worker-pool3-t9] master.RegionStates(1112): Transition {32117afaa6ee9f522aba3ba6ee438665 state=PENDING_CLOSE, ts=1428169689084, server=localhost,51330,1428169644428} to {32117afaa6ee9f522aba3ba6ee438665 state=OFFLINE, ts=1428169689130, server=localhost,51330,1428169644428} 2015-04-04 10:48:09,130 INFO [AM.ZK.Worker-pool3-t9] master.RegionStates(600): Offlined 32117afaa6ee9f522aba3ba6ee438665 from localhost,51330,1428169644428 2015-04-04 10:48:09,138 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testIncrementHook state from DISABLING to DISABLED 2015-04-04 10:48:09,139 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(191): Disabled table, TestTable.testIncrementHook, is done=true 2015-04-04 10:48:09,141 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testIncrementHook/write-master:513270000000001 2015-04-04 10:48:09,199 INFO [Thread-532] client.HBaseAdmin(1034): Disabled TestTable.testIncrementHook 2015-04-04 10:48:09,200 INFO [B.defaultRpcServer.handler=4,queue=0,port=51327] master.HMaster(1564): Client=apurtell/null delete TestTable.testIncrementHook 2015-04-04 10:48:09,202 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testIncrementHook/write-master:513270000000002 2015-04-04 10:48:09,208 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table TestTable.testIncrementHook 2015-04-04 10:48:09,212 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:48:09,214 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => 32117afaa6ee9f522aba3ba6ee438665, NAME => 'TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:48:09,216 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(186): Archiving region TestTable.testIncrementHook,,1428169688476.32117afaa6ee9f522aba3ba6ee438665. from FS 2015-04-04 10:48:09,216 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665 2015-04-04 10:48:09,221 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/recovered.edits] 2015-04-04 10:48:09,229 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/a/2020a03eefba4920843651cf4e920257, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/a/2020a03eefba4920843651cf4e920257 2015-04-04 10:48:09,239 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/recovered.edits/9.seqid, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665/recovered.edits/9.seqid 2015-04-04 10:48:09,240 INFO [IPC Server handler 9 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741883_1059 127.0.0.1:51320 2015-04-04 10:48:09,241 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testIncrementHook/32117afaa6ee9f522aba3ba6ee438665 2015-04-04 10:48:09,242 INFO [IPC Server handler 4 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741882_1058 127.0.0.1:51320 2015-04-04 10:48:09,242 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(196): Table 'TestTable.testIncrementHook' archived! 2015-04-04 10:48:09,242 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(116): Removing 'TestTable.testIncrementHook' descriptor. 2015-04-04 10:48:09,243 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(122): Removing 'TestTable.testIncrementHook' from region states. 2015-04-04 10:48:09,243 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(126): Marking 'TestTable.testIncrementHook' as deleted. 2015-04-04 10:48:09,248 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testIncrementHook/write-master:513270000000002 2015-04-04 10:48:09,323 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=51327] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = TestTable.testIncrementHook org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testIncrementHook at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:48:09,323 INFO [Thread-532] client.HBaseAdmin(739): Deleted TestTable.testIncrementHook 2015-04-04 10:48:09,354 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testIncrementHook Thread=570 (was 551) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t154 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t153 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51416 [Waiting for operation #35] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t145 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t119 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1796537615_1 at /127.0.0.1:51459 [Waiting for operation #9] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t122 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t117 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x34c6ce8e-shared--pool9-t16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51417 [Waiting for operation #35] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t156 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t149 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t152 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t120 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t118 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t150 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t148 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t155 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t146 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t147 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t151 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t121 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=435 (was 435), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=232 (was 232), ProcessCount=232 (was 232), AvailableMemoryMB=16 (was 16), ConnectionCount=1 (was 1) 2015-04-04 10:48:09,354 WARN [main] hbase.ResourceChecker(134): Thread=570 is superior to 500 2015-04-04 10:48:09,385 INFO [main] hbase.ResourceChecker(147): before: coprocessor.TestRegionObserverInterface#testAppendHook Thread=570, OpenFileDescriptor=435, MaxFileDescriptor=10240, SystemLoadAverage=232, ProcessCount=232, AvailableMemoryMB=16, ConnectionCount=1 2015-04-04 10:48:09,385 WARN [main] hbase.ResourceChecker(134): Thread=570 is superior to 500 2015-04-04 10:48:09,387 INFO [B.defaultRpcServer.handler=3,queue=0,port=51327] master.HMaster(1318): Client=apurtell/null create 'TestTable.testAppendHook', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} 2015-04-04 10:48:09,401 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testAppendHook/write-master:513270000000000 2015-04-04 10:48:09,409 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(188): Create table TestTable.testAppendHook 2015-04-04 10:48:09,419 INFO [IPC Server handler 2 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741885_1061{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 0 2015-04-04 10:48:09,422 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testAppendHook/.tabledesc/.tableinfo.0000000001 2015-04-04 10:48:09,424 INFO [RegionOpenAndInitThread-TestTable.testAppendHook-1] regionserver.HRegion(5885): creating HRegion TestTable.testAppendHook HTD == 'TestTable.testAppendHook', {NAME => 'a', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'b', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp Table name == TestTable.testAppendHook 2015-04-04 10:48:09,432 INFO [IPC Server handler 2 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741886_1062{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 0 2015-04-04 10:48:09,433 DEBUG [RegionOpenAndInitThread-TestTable.testAppendHook-1] regionserver.HRegion(721): Instantiated TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. 2015-04-04 10:48:09,433 DEBUG [RegionOpenAndInitThread-TestTable.testAppendHook-1] regionserver.HRegion(1350): Closing TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764.: disabling compactions & flushes 2015-04-04 10:48:09,433 DEBUG [RegionOpenAndInitThread-TestTable.testAppendHook-1] regionserver.HRegion(1377): Updates disabled for region TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. 2015-04-04 10:48:09,433 INFO [RegionOpenAndInitThread-TestTable.testAppendHook-1] regionserver.HRegion(1471): Closed TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. 2015-04-04 10:48:09,437 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1169): Added 2 2015-04-04 10:48:09,438 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1605): Assigning 1 region(s) to localhost,51330,1428169644428 2015-04-04 10:48:09,438 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKAssign(174): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Async create of unassigned node 66f1a885119863900aed5e146f605764 with OFFLINE state 2015-04-04 10:48:09,440 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:09,440 DEBUG [main-EventThread] master.OfflineCallback(69): rs={66f1a885119863900aed5e146f605764 state=OFFLINE, ts=1428169689438, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:48:09,441 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={66f1a885119863900aed5e146f605764 state=OFFLINE, ts=1428169689438, server=null}, server=localhost,51330,1428169644428 2015-04-04 10:48:09,445 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1658): localhost,51330,1428169644428 unassigned znodes=1 of total=1; oldCounter=0 2015-04-04 10:48:09,445 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.RegionStates(1112): Transition {66f1a885119863900aed5e146f605764 state=OFFLINE, ts=1428169689438, server=null} to {66f1a885119863900aed5e146f605764 state=PENDING_OPEN, ts=1428169689445, server=localhost,51330,1428169644428} 2015-04-04 10:48:09,446 INFO [PriorityRpcServer.handler=8,queue=0,port=51330] regionserver.RSRpcServices(1390): Open TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. 2015-04-04 10:48:09,456 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning 66f1a885119863900aed5e146f605764 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:48:09,456 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] master.AssignmentManager(1788): Bulk assigning done for localhost,51330,1428169644428 2015-04-04 10:48:09,457 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testAppendHook state from ENABLING to ENABLED 2015-04-04 10:48:09,458 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/66f1a885119863900aed5e146f605764 2015-04-04 10:48:09,458 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node 66f1a885119863900aed5e146f605764 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2015-04-04 10:48:09,459 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(6078): Opening region: {ENCODED => 66f1a885119863900aed5e146f605764, NAME => 'TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:48:09,459 DEBUG [AM.ZK.Worker-pool3-t8] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENING, server=localhost,51330,1428169644428, region=66f1a885119863900aed5e146f605764, current_state={66f1a885119863900aed5e146f605764 state=PENDING_OPEN, ts=1428169689445, server=localhost,51330,1428169644428} 2015-04-04 10:48:09,459 INFO [AM.ZK.Worker-pool3-t8] master.RegionStates(1112): Transition {66f1a885119863900aed5e146f605764 state=PENDING_OPEN, ts=1428169689445, server=localhost,51330,1428169644428} to {66f1a885119863900aed5e146f605764 state=OPENING, ts=1428169689459, server=localhost,51330,1428169644428} 2015-04-04 10:48:09,459 DEBUG [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(599): Found an implementation of 'preWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:48:09,459 DEBUG [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(599): Found an implementation of 'postWALRestore' that uses updated method signature. Skipping legacy support for invocations in 'class org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver'. 2015-04-04 10:48:09,460 INFO [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver was loaded successfully with priority (536870911). 2015-04-04 10:48:09,460 INFO [RS_OPEN_REGION-localhost:51330-1] coprocessor.CoprocessorHost(157): System coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy was loaded successfully with priority (536870912). 2015-04-04 10:48:09,460 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.MetricsRegionSourceImpl(67): Creating new MetricsRegionSourceImpl for table TestTable.testAppendHook 66f1a885119863900aed5e146f605764 2015-04-04 10:48:09,460 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(721): Instantiated TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. 2015-04-04 10:48:09,462 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testAppendHook/write-master:513270000000000 2015-04-04 10:48:09,462 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.CreateTableHandler(218): Table, TestTable.testAppendHook, creation successful 2015-04-04 10:48:09,463 INFO [StoreOpener-66f1a885119863900aed5e146f605764-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:09,464 INFO [StoreOpener-66f1a885119863900aed5e146f605764-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:09,465 DEBUG [StoreOpener-66f1a885119863900aed5e146f605764-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/a 2015-04-04 10:48:09,467 INFO [StoreOpener-66f1a885119863900aed5e146f605764-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:09,467 INFO [StoreOpener-66f1a885119863900aed5e146f605764-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:09,468 DEBUG [StoreOpener-66f1a885119863900aed5e146f605764-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/b 2015-04-04 10:48:09,470 INFO [StoreOpener-66f1a885119863900aed5e146f605764-1] hfile.CacheConfig(278): blockCache=LruBlockCache{blockCount=1, currentSize=1072608, freeSize=1042889696, maxSize=1043962304, heapSize=1072608, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2015-04-04 10:48:09,470 INFO [StoreOpener-66f1a885119863900aed5e146f605764-1] compactions.CompactionConfiguration(97): size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2015-04-04 10:48:09,472 DEBUG [StoreOpener-66f1a885119863900aed5e146f605764-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/c 2015-04-04 10:48:09,472 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(3756): Found 0 recovered edits file(s) under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764 2015-04-04 10:48:09,474 DEBUG [RS_OPEN_REGION-localhost:51330-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2015-04-04 10:48:09,478 DEBUG [RS_OPEN_REGION-localhost:51330-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2015-04-04 10:48:09,478 INFO [RS_OPEN_REGION-localhost:51330-1] regionserver.HRegion(847): Onlined 66f1a885119863900aed5e146f605764; next sequenceid=2 2015-04-04 10:48:09,479 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(643): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Attempting to retransition opening state of node 66f1a885119863900aed5e146f605764 2015-04-04 10:48:09,480 INFO [PostOpenDeployTasks:66f1a885119863900aed5e146f605764] regionserver.HRegionServer(1849): Post open deploy tasks for TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. 2015-04-04 10:48:09,483 INFO [PostOpenDeployTasks:66f1a885119863900aed5e146f605764] hbase.MetaTableAccessor(1350): Updated row TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. with server=localhost,51330,1428169644428 2015-04-04 10:48:09,483 DEBUG [PostOpenDeployTasks:66f1a885119863900aed5e146f605764] regionserver.HRegionServer(1883): Finished post open deploy task for TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. 2015-04-04 10:48:09,486 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning 66f1a885119863900aed5e146f605764 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:48:09,488 DEBUG [RS_OPEN_REGION-localhost:51330-1] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node 66f1a885119863900aed5e146f605764 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2015-04-04 10:48:09,488 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/66f1a885119863900aed5e146f605764 2015-04-04 10:48:09,488 DEBUG [RS_OPEN_REGION-localhost:51330-1] coordination.ZkOpenRegionCoordination(91): Transitioned 66f1a885119863900aed5e146f605764 to OPENED in zk on localhost,51330,1428169644428 2015-04-04 10:48:09,488 DEBUG [RS_OPEN_REGION-localhost:51330-1] handler.OpenRegionHandler(167): Opened TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. on localhost,51330,1428169644428 2015-04-04 10:48:09,488 DEBUG [AM.ZK.Worker-pool3-t12] master.AssignmentManager(948): Handling RS_ZK_REGION_OPENED, server=localhost,51330,1428169644428, region=66f1a885119863900aed5e146f605764, current_state={66f1a885119863900aed5e146f605764 state=OPENING, ts=1428169689459, server=localhost,51330,1428169644428} 2015-04-04 10:48:09,488 INFO [AM.ZK.Worker-pool3-t12] master.RegionStates(1112): Transition {66f1a885119863900aed5e146f605764 state=OPENING, ts=1428169689459, server=localhost,51330,1428169644428} to {66f1a885119863900aed5e146f605764 state=OPEN, ts=1428169689488, server=localhost,51330,1428169644428} 2015-04-04 10:48:09,488 DEBUG [AM.ZK.Worker-pool3-t12] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 66f1a885119863900aed5e146f605764 from localhost,51327,1428169644129; deleting unassigned node 2015-04-04 10:48:09,489 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/66f1a885119863900aed5e146f605764 2015-04-04 10:48:09,490 DEBUG [AM.ZK.Worker-pool3-t12] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node 66f1a885119863900aed5e146f605764 in expected state RS_ZK_REGION_OPENED 2015-04-04 10:48:09,490 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:09,490 DEBUG [AM.ZK.Worker-pool3-t12] master.AssignmentManager$4(1337): Znode TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. deleted, state: {66f1a885119863900aed5e146f605764 state=OPEN, ts=1428169689488, server=localhost,51330,1428169644428} 2015-04-04 10:48:09,490 DEBUG [AM.ZK.Worker-pool3-t12] master.RegionStates(460): Onlined 66f1a885119863900aed5e146f605764 on localhost,51330,1428169644428 {ENCODED => 66f1a885119863900aed5e146f605764, NAME => 'TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764.', STARTKEY => '', ENDKEY => ''} 2015-04-04 10:48:09,528 INFO [Thread-547] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:48:09,533 INFO [Thread-547] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2015-04-04 10:48:09,539 INFO [Thread-547] client.HBaseAdmin$9(979): Started disable of TestTable.testAppendHook 2015-04-04 10:48:09,539 INFO [B.defaultRpcServer.handler=0,queue=0,port=51327] master.HMaster(1659): Client=apurtell/null disable TestTable.testAppendHook 2015-04-04 10:48:09,541 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testAppendHook/write-master:513270000000001 2015-04-04 10:48:09,547 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(129): Attempting to disable table TestTable.testAppendHook 2015-04-04 10:48:09,547 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testAppendHook state from DISABLING to DISABLING 2015-04-04 10:48:09,549 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(174): Offlining 1 regions. 2015-04-04 10:48:09,549 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(2452): Starting unassign of TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. (offlining), current state: {66f1a885119863900aed5e146f605764 state=OPEN, ts=1428169689490, server=localhost,51330,1428169644428} 2015-04-04 10:48:09,549 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] zookeeper.ZKAssign(527): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Creating unassigned node 66f1a885119863900aed5e146f605764 in a CLOSING state 2015-04-04 10:48:09,550 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:09,551 INFO [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.RegionStates(1112): Transition {66f1a885119863900aed5e146f605764 state=OPEN, ts=1428169689490, server=localhost,51330,1428169644428} to {66f1a885119863900aed5e146f605764 state=PENDING_CLOSE, ts=1428169689551, server=localhost,51330,1428169644428} 2015-04-04 10:48:09,551 INFO [PriorityRpcServer.handler=6,queue=0,port=51330] regionserver.RSRpcServices(1018): Close 66f1a885119863900aed5e146f605764, moving to null 2015-04-04 10:48:09,551 DEBUG [RS_CLOSE_REGION-localhost:51330-2] handler.CloseRegionHandler(119): Processing close of TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. 2015-04-04 10:48:09,551 DEBUG [localhost,51327,1428169644129-org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler-0] master.AssignmentManager(1836): Sent CLOSE to localhost,51330,1428169644428 for region TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. 2015-04-04 10:48:09,552 DEBUG [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegion(1350): Closing TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764.: disabling compactions & flushes 2015-04-04 10:48:09,552 DEBUG [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegion(1377): Updates disabled for region TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. 2015-04-04 10:48:09,553 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegion(2084): Started memstore flush for TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764., current region memstore size 160 B, and 3/3 column families' memstores are being flushed. 2015-04-04 10:48:09,554 DEBUG [Thread-547] client.HBaseAdmin(1019): Sleeping= 100ms, waiting for all regions to be disabled in TestTable.testAppendHook 2015-04-04 10:48:09,563 INFO [IPC Server handler 2 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741887_1063{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 4718 2015-04-04 10:48:09,668 DEBUG [Thread-547] client.HBaseAdmin(1019): Sleeping= 200ms, waiting for all regions to be disabled in TestTable.testAppendHook 2015-04-04 10:48:09,877 DEBUG [Thread-547] client.HBaseAdmin(1019): Sleeping= 300ms, waiting for all regions to be disabled in TestTable.testAppendHook 2015-04-04 10:48:09,967 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=6, memsize=160, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/.tmp/67a96ab99d2f4104907e908214a37e96 2015-04-04 10:48:09,977 DEBUG [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/.tmp/67a96ab99d2f4104907e908214a37e96 as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/a/67a96ab99d2f4104907e908214a37e96 2015-04-04 10:48:09,986 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/a/67a96ab99d2f4104907e908214a37e96, entries=1, sequenceid=6, filesize=4.6 K 2015-04-04 10:48:09,987 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegion(2363): Finished memstore flush of ~160 B/160, currentsize=0 B/0 for region TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. in 434ms, sequenceid=6, compaction requested=false 2015-04-04 10:48:09,989 INFO [StoreCloserThread-TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764.-1] regionserver.HStore(879): Closed a 2015-04-04 10:48:09,989 INFO [StoreCloserThread-TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764.-1] regionserver.HStore(879): Closed b 2015-04-04 10:48:09,989 INFO [StoreCloserThread-TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764.-1] regionserver.HStore(879): Closed c 2015-04-04 10:48:09,997 DEBUG [RS_CLOSE_REGION-localhost:51330-2] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/recovered.edits/9.seqid to file, newSeqId=9, maxSeqId=2 2015-04-04 10:48:09,998 DEBUG [RS_CLOSE_REGION-localhost:51330-2] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:48:09,998 DEBUG [RS_CLOSE_REGION-localhost:51330-2] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:48:09,998 INFO [RS_CLOSE_REGION-localhost:51330-2] regionserver.HRegion(1471): Closed TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. 2015-04-04 10:48:09,999 DEBUG [RS_CLOSE_REGION-localhost:51330-2] zookeeper.ZKAssign(805): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioning 66f1a885119863900aed5e146f605764 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:48:10,008 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/66f1a885119863900aed5e146f605764 2015-04-04 10:48:10,008 DEBUG [RS_CLOSE_REGION-localhost:51330-2] zookeeper.ZKAssign(880): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Transitioned node 66f1a885119863900aed5e146f605764 from M_ZK_REGION_CLOSING to RS_ZK_REGION_CLOSED 2015-04-04 10:48:10,009 DEBUG [RS_CLOSE_REGION-localhost:51330-2] coordination.ZkCloseRegionCoordination(79): Set closed state in zk for TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. on localhost,51330,1428169644428 2015-04-04 10:48:10,009 DEBUG [RS_CLOSE_REGION-localhost:51330-2] handler.CloseRegionHandler(164): Closed TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. 2015-04-04 10:48:10,009 DEBUG [AM.ZK.Worker-pool3-t19] master.AssignmentManager(948): Handling RS_ZK_REGION_CLOSED, server=localhost,51330,1428169644428, region=66f1a885119863900aed5e146f605764, current_state={66f1a885119863900aed5e146f605764 state=PENDING_CLOSE, ts=1428169689551, server=localhost,51330,1428169644428} 2015-04-04 10:48:10,009 DEBUG [AM.ZK.Worker-pool3-t19] handler.ClosedRegionHandler(93): Handling CLOSED event for 66f1a885119863900aed5e146f605764 2015-04-04 10:48:10,009 DEBUG [AM.ZK.Worker-pool3-t19] master.AssignmentManager(1530): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. 2015-04-04 10:48:10,011 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/66f1a885119863900aed5e146f605764 2015-04-04 10:48:10,011 DEBUG [AM.ZK.Worker-pool3-t19] zookeeper.ZKAssign(479): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Deleted unassigned node 66f1a885119863900aed5e146f605764 in expected state RS_ZK_REGION_CLOSED 2015-04-04 10:48:10,011 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2015-04-04 10:48:10,012 INFO [AM.ZK.Worker-pool3-t19] master.RegionStates(1112): Transition {66f1a885119863900aed5e146f605764 state=PENDING_CLOSE, ts=1428169689551, server=localhost,51330,1428169644428} to {66f1a885119863900aed5e146f605764 state=OFFLINE, ts=1428169690011, server=localhost,51330,1428169644428} 2015-04-04 10:48:10,012 INFO [AM.ZK.Worker-pool3-t19] master.RegionStates(600): Offlined 66f1a885119863900aed5e146f605764 from localhost,51330,1428169644428 2015-04-04 10:48:10,028 WARN [MASTER_TABLE_OPERATIONS-localhost:51327-0] zookeeper.ZKTableStateManager(100): Moving table TestTable.testAppendHook state from DISABLING to DISABLED 2015-04-04 10:48:10,029 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DisableTableHandler(191): Disabled table, TestTable.testAppendHook, is done=true 2015-04-04 10:48:10,030 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testAppendHook/write-master:513270000000001 2015-04-04 10:48:10,190 INFO [Thread-547] client.HBaseAdmin(1034): Disabled TestTable.testAppendHook 2015-04-04 10:48:10,191 INFO [B.defaultRpcServer.handler=3,queue=0,port=51327] master.HMaster(1564): Client=apurtell/null delete TestTable.testAppendHook 2015-04-04 10:48:10,201 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=51327] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestTable.testAppendHook/write-master:513270000000002 2015-04-04 10:48:10,208 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.TableEventHandler(126): Handling table operation C_M_DELETE_TABLE on table TestTable.testAppendHook 2015-04-04 10:48:10,214 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(171): Deleting regions from META 2015-04-04 10:48:10,216 INFO [MASTER_TABLE_OPERATIONS-localhost:51327-0] hbase.MetaTableAccessor(1381): Deleted [{ENCODED => 66f1a885119863900aed5e146f605764, NAME => 'TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764.', STARTKEY => '', ENDKEY => ''}] 2015-04-04 10:48:10,218 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(186): Archiving region TestTable.testAppendHook,,1428169689387.66f1a885119863900aed5e146f605764. from FS 2015-04-04 10:48:10,218 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764 2015-04-04 10:48:10,222 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/a, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/b, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/c, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/recovered.edits] 2015-04-04 10:48:10,230 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/a/67a96ab99d2f4104907e908214a37e96, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/a/67a96ab99d2f4104907e908214a37e96 2015-04-04 10:48:10,240 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/recovered.edits/9.seqid, to hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/archive/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764/recovered.edits/9.seqid 2015-04-04 10:48:10,241 INFO [IPC Server handler 0 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741886_1062 127.0.0.1:51320 2015-04-04 10:48:10,241 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/.tmp/data/default/TestTable.testAppendHook/66f1a885119863900aed5e146f605764 2015-04-04 10:48:10,242 INFO [IPC Server handler 8 on 51319] blockmanagement.BlockManager(1074): BLOCK* addToInvalidates: blk_1073741885_1061 127.0.0.1:51320 2015-04-04 10:48:10,242 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(196): Table 'TestTable.testAppendHook' archived! 2015-04-04 10:48:10,242 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(116): Removing 'TestTable.testAppendHook' descriptor. 2015-04-04 10:48:10,243 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(122): Removing 'TestTable.testAppendHook' from region states. 2015-04-04 10:48:10,243 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] handler.DeleteTableHandler(126): Marking 'TestTable.testAppendHook' as deleted. 2015-04-04 10:48:10,249 DEBUG [MASTER_TABLE_OPERATIONS-localhost:51327-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestTable.testAppendHook/write-master:513270000000002 2015-04-04 10:48:10,324 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=51327] util.FSTableDescriptors(177): Exception during readTableDecriptor. Current table name = TestTable.testAppendHook org.apache.hadoop.hbase.TableInfoMissingException: No table descriptor file under hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestTable.testAppendHook at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:509) at org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:487) at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:172) at org.apache.hadoop.hbase.master.HMaster.listTableDescriptors(HMaster.java:2215) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:798) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44807) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2093) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-04-04 10:48:10,325 INFO [Thread-547] client.HBaseAdmin(739): Deleted TestTable.testAppendHook 2015-04-04 10:48:10,366 INFO [main] hbase.ResourceChecker(171): after: coprocessor.TestRegionObserverInterface#testAppendHook Thread=593 (was 570) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t128 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51416 [Waiting for operation #37] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t160 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t125 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t164 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t163 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1796537615_1 at /127.0.0.1:51459 [Waiting for operation #11] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t131 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t126 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t159 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t167 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t161 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_429382542_1 at /127.0.0.1:51417 [Waiting for operation #37] sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read(BufferedInputStream.java:254) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:55) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:211) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t168 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t166 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t129 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t162 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t132 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t127 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t157 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t158 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t130 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t123 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x3b4ad720-shared--pool16-t124 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x6e9fabc7-shared--pool7-t165 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x34c6ce8e-shared--pool9-t17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=435 (was 435), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=232 (was 232), ProcessCount=232 (was 232), AvailableMemoryMB=16 (was 16), ConnectionCount=1 (was 1) 2015-04-04 10:48:10,367 WARN [main] hbase.ResourceChecker(134): Thread=593 is superior to 500 2015-04-04 10:48:10,367 INFO [main] hbase.HBaseTestingUtility(1049): Shutting down minicluster 2015-04-04 10:48:10,367 INFO [main] client.ConnectionManager$HConnectionImplementation(2055): Closing master protocol: MasterService 2015-04-04 10:48:10,373 INFO [main] client.ConnectionManager$HConnectionImplementation(1675): Closing zookeeper sessionid=0x14c858b55e60008 2015-04-04 10:48:10,382 DEBUG [main] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:48:10,383 DEBUG [RpcServer.reader=0,bindAddress=localhost,port=51330] ipc.RpcServer$Listener(861): RpcServer.listener,port=51330: DISCONNECTING client 127.0.0.1:51449 because read count=-1. Number of active connections: 6 2015-04-04 10:48:10,383 DEBUG [RpcServer.reader=2,bindAddress=localhost,port=51327] ipc.RpcServer$Listener(861): RpcServer.listener,port=51327: DISCONNECTING client 127.0.0.1:51356 because read count=-1. Number of active connections: 2 2015-04-04 10:48:10,383 DEBUG [RpcServer.reader=5,bindAddress=localhost,port=51330] ipc.RpcServer$Listener(861): RpcServer.listener,port=51330: DISCONNECTING client 127.0.0.1:51357 because read count=-1. Number of active connections: 5 2015-04-04 10:48:10,486 DEBUG [main] util.JVMClusterUtil(241): Shutting down HBase Cluster 2015-04-04 10:48:10,486 INFO [main] regionserver.HRegionServer(1823): STOPPED: Cluster shutdown requested 2015-04-04 10:48:10,487 INFO [M:0;localhost:51327] regionserver.HRegionServer(955): Stopping infoServer 2015-04-04 10:48:10,495 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2015-04-04 10:48:10,495 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2015-04-04 10:48:10,495 INFO [main] regionserver.HRegionServer(1823): STOPPED: Shutdown requested 2015-04-04 10:48:10,496 INFO [RS:0;localhost:51330] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2015-04-04 10:48:10,496 INFO [RS:0;localhost:51330] regionserver.HRegionServer(955): Stopping infoServer 2015-04-04 10:48:10,496 DEBUG [main-EventThread] zookeeper.ZKUtil(492): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2015-04-04 10:48:10,496 DEBUG [main-EventThread] zookeeper.ZKUtil(492): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2015-04-04 10:48:10,496 INFO [SplitLogWorker-localhost:51330] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2015-04-04 10:48:10,496 INFO [M:0;localhost:51327] log.Slf4jLog(67): Stopped SelectChannelConnector@0.0.0.0:0 2015-04-04 10:48:10,499 INFO [SplitLogWorker-localhost:51330] regionserver.SplitLogWorker(155): SplitLogWorker localhost,51330,1428169644428 exiting 2015-04-04 10:48:10,502 INFO [RS:0;localhost:51330] log.Slf4jLog(67): Stopped SelectChannelConnector@0.0.0.0:0 2015-04-04 10:48:10,605 INFO [M:0;localhost:51327] regionserver.HRegionServer(995): stopping server localhost,51327,1428169644129 2015-04-04 10:48:10,605 DEBUG [M:0;localhost:51327] zookeeper.MetaTableLocator(619): Stopping MetaTableLocator 2015-04-04 10:48:10,607 INFO [RS:0;localhost:51330] regionserver.HeapMemoryManager(198): Stoping HeapMemoryTuner chore. 2015-04-04 10:48:10,607 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2015-04-04 10:48:10,607 INFO [RS:0;localhost:51330] snapshot.RegionServerSnapshotManager(134): Stopping RegionServerSnapshotManager gracefully. 2015-04-04 10:48:10,608 INFO [RS:0;localhost:51330] flush.RegionServerFlushTableProcedureManager(113): Stopping region server flush procedure manager gracefully. 2015-04-04 10:48:10,608 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2015-04-04 10:48:10,609 DEBUG [RS_CLOSE_REGION-localhost:51330-0] handler.CloseRegionHandler(119): Processing close of TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. 2015-04-04 10:48:10,609 DEBUG [RS_CLOSE_REGION-localhost:51330-1] handler.CloseRegionHandler(119): Processing close of hbase:namespace,,1428169646774.17aa502c05dd762f269eb05ccfc542db. 2015-04-04 10:48:10,609 INFO [RS:0;localhost:51330] regionserver.HRegionServer(995): stopping server localhost,51330,1428169644428 2015-04-04 10:48:10,610 DEBUG [RS:0;localhost:51330] zookeeper.MetaTableLocator(619): Stopping MetaTableLocator 2015-04-04 10:48:10,610 DEBUG [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(1350): Closing hbase:namespace,,1428169646774.17aa502c05dd762f269eb05ccfc542db.: disabling compactions & flushes 2015-04-04 10:48:10,611 DEBUG [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(1377): Updates disabled for region hbase:namespace,,1428169646774.17aa502c05dd762f269eb05ccfc542db. 2015-04-04 10:48:10,609 DEBUG [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(1350): Closing TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46.: disabling compactions & flushes 2015-04-04 10:48:10,611 DEBUG [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(1377): Updates disabled for region TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. 2015-04-04 10:48:10,611 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(2084): Started memstore flush for hbase:namespace,,1428169646774.17aa502c05dd762f269eb05ccfc542db., current region memstore size 344 B, and 1/1 column families' memstores are being flushed. 2015-04-04 10:48:10,613 INFO [StoreCloserThread-TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46.-1] regionserver.HStore(879): Closed a 2015-04-04 10:48:10,613 INFO [RS:0;localhost:51330] client.ConnectionManager$HConnectionImplementation(1675): Closing zookeeper sessionid=0x14c858b55e60003 2015-04-04 10:48:10,620 INFO [M:0;localhost:51327] client.ConnectionManager$HConnectionImplementation(1675): Closing zookeeper sessionid=0x14c858b55e60002 2015-04-04 10:48:10,622 DEBUG [RS_CLOSE_REGION-localhost:51330-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/default/TestCompactionOverride/1d8e5c1a351aa112c824cf548d63bb46/recovered.edits/28.seqid to file, newSeqId=28, maxSeqId=2 2015-04-04 10:48:10,622 DEBUG [RS:0;localhost:51330] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:48:10,622 DEBUG [M:0;localhost:51327] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:48:10,623 DEBUG [RpcServer.reader=3,bindAddress=localhost,port=51330] ipc.RpcServer$Listener(861): RpcServer.listener,port=51330: DISCONNECTING client 127.0.0.1:51352 because read count=-1. Number of active connections: 4 2015-04-04 10:48:10,623 DEBUG [RpcServer.reader=2,bindAddress=localhost,port=51330] ipc.RpcServer$Listener(861): RpcServer.listener,port=51330: DISCONNECTING client 127.0.0.1:51349 because read count=-1. Number of active connections: 4 2015-04-04 10:48:10,623 DEBUG [RS_CLOSE_REGION-localhost:51330-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:48:10,623 DEBUG [RS_CLOSE_REGION-localhost:51330-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:48:10,623 DEBUG [RS_CLOSE_REGION-localhost:51330-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface$EvenOnlyCompactor 2015-04-04 10:48:10,624 INFO [RS_CLOSE_REGION-localhost:51330-0] regionserver.HRegion(1471): Closed TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. 2015-04-04 10:48:10,624 DEBUG [RS_CLOSE_REGION-localhost:51330-0] handler.CloseRegionHandler(164): Closed TestCompactionOverride,,1428169681939.1d8e5c1a351aa112c824cf548d63bb46. 2015-04-04 10:48:10,625 INFO [IPC Server handler 1 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741888_1064{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 0 2015-04-04 10:48:10,626 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=8, memsize=344, hasBloomFilter=true, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/hbase/namespace/17aa502c05dd762f269eb05ccfc542db/.tmp/86bb03bb7daf46eeaece324c86bb8698 2015-04-04 10:48:10,634 DEBUG [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/hbase/namespace/17aa502c05dd762f269eb05ccfc542db/.tmp/86bb03bb7daf46eeaece324c86bb8698 as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/hbase/namespace/17aa502c05dd762f269eb05ccfc542db/info/86bb03bb7daf46eeaece324c86bb8698 2015-04-04 10:48:10,642 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/hbase/namespace/17aa502c05dd762f269eb05ccfc542db/info/86bb03bb7daf46eeaece324c86bb8698, entries=2, sequenceid=8, filesize=4.9 K 2015-04-04 10:48:10,643 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(2363): Finished memstore flush of ~344 B/344, currentsize=0 B/0 for region hbase:namespace,,1428169646774.17aa502c05dd762f269eb05ccfc542db. in 32ms, sequenceid=8, compaction requested=false 2015-04-04 10:48:10,645 INFO [StoreCloserThread-hbase:namespace,,1428169646774.17aa502c05dd762f269eb05ccfc542db.-1] regionserver.HStore(879): Closed info 2015-04-04 10:48:10,652 DEBUG [RS_CLOSE_REGION-localhost:51330-1] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/hbase/namespace/17aa502c05dd762f269eb05ccfc542db/recovered.edits/11.seqid to file, newSeqId=11, maxSeqId=2 2015-04-04 10:48:10,653 INFO [localhost,51330,1428169644428_ChoreService_1] hbase.ScheduledChore(179): Chore: localhost,51330,1428169644428-MemstoreFlusherChore was stopped 2015-04-04 10:48:10,653 DEBUG [RS_CLOSE_REGION-localhost:51330-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:48:10,653 DEBUG [RS_CLOSE_REGION-localhost:51330-1] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:48:10,653 INFO [RS_CLOSE_REGION-localhost:51330-1] regionserver.HRegion(1471): Closed hbase:namespace,,1428169646774.17aa502c05dd762f269eb05ccfc542db. 2015-04-04 10:48:10,653 DEBUG [RS_CLOSE_REGION-localhost:51330-1] handler.CloseRegionHandler(164): Closed hbase:namespace,,1428169646774.17aa502c05dd762f269eb05ccfc542db. 2015-04-04 10:48:10,725 INFO [RS:0;localhost:51330] regionserver.CompactSplitThread(393): Waiting for Split Thread to finish... 2015-04-04 10:48:10,725 INFO [M:0;localhost:51327] regionserver.HRegionServer(1023): stopping server localhost,51327,1428169644129; all regions closed. 2015-04-04 10:48:10,725 INFO [RS:0;localhost:51330] regionserver.CompactSplitThread(393): Waiting for Merge Thread to finish... 2015-04-04 10:48:10,725 INFO [RS:0;localhost:51330] regionserver.CompactSplitThread(393): Waiting for Large Compaction Thread to finish... 2015-04-04 10:48:10,725 INFO [RS:0;localhost:51330] regionserver.CompactSplitThread(393): Waiting for Small Compaction Thread to finish... 2015-04-04 10:48:10,725 INFO [M:0;localhost:51327] hbase.ChoreService(303): Chore service for: localhost,51327,1428169644129 had [java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@4f26b047, java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@74ff2de9, java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1d1f5801, java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@74cab896, java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@291cbbc1] on shutdown 2015-04-04 10:48:10,726 INFO [RS:0;localhost:51330] regionserver.HRegionServer(1217): Waiting on 1 regions to close 2015-04-04 10:48:10,726 DEBUG [RS:0;localhost:51330] regionserver.HRegionServer(1221): {1588230740=hbase:meta,,1.1588230740} 2015-04-04 10:48:10,726 DEBUG [RS_CLOSE_META-localhost:51330-0] handler.CloseRegionHandler(119): Processing close of hbase:meta,,1.1588230740 2015-04-04 10:48:10,726 INFO [M:0;localhost:51327] master.ServerManager(546): Waiting on regionserver(s) to go down localhost,51330,1428169644428 2015-04-04 10:48:10,727 DEBUG [RS_CLOSE_META-localhost:51330-0] regionserver.HRegion(1350): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2015-04-04 10:48:10,727 DEBUG [RS_CLOSE_META-localhost:51330-0] regionserver.HRegion(1377): Updates disabled for region hbase:meta,,1.1588230740 2015-04-04 10:48:10,728 INFO [RS_CLOSE_META-localhost:51330-0] regionserver.HRegion(2084): Started memstore flush for hbase:meta,,1.1588230740, current region memstore size 21.44 KB, and 1/1 column families' memstores are being flushed. 2015-04-04 10:48:10,740 INFO [IPC Server handler 2 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741889_1065{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 0 2015-04-04 10:48:10,741 INFO [RS_CLOSE_META-localhost:51330-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=97, memsize=21.4 K, hasBloomFilter=false, into tmp file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/hbase/meta/1588230740/.tmp/31e1ebf9038540f893512672c2c65ac3 2015-04-04 10:48:10,750 INFO [RS_CLOSE_META-localhost:51330-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 31e1ebf9038540f893512672c2c65ac3 2015-04-04 10:48:10,751 DEBUG [RS_CLOSE_META-localhost:51330-0] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/hbase/meta/1588230740/.tmp/31e1ebf9038540f893512672c2c65ac3 as hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/hbase/meta/1588230740/info/31e1ebf9038540f893512672c2c65ac3 2015-04-04 10:48:10,758 INFO [RS_CLOSE_META-localhost:51330-0] regionserver.StoreFile$Reader(1404): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 31e1ebf9038540f893512672c2c65ac3 2015-04-04 10:48:10,758 INFO [RS_CLOSE_META-localhost:51330-0] regionserver.HStore(973): Added hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/hbase/meta/1588230740/info/31e1ebf9038540f893512672c2c65ac3, entries=20, sequenceid=97, filesize=7.1 K 2015-04-04 10:48:10,759 INFO [RS_CLOSE_META-localhost:51330-0] regionserver.HRegion(2363): Finished memstore flush of ~21.44 KB/21952, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 31ms, sequenceid=97, compaction requested=false 2015-04-04 10:48:10,761 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(879): Closed info 2015-04-04 10:48:10,767 DEBUG [RS_CLOSE_META-localhost:51330-0] wal.WALSplitter(720): Wrote region seqId=hdfs://localhost:51319/user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/data/hbase/meta/1588230740/recovered.edits/100.seqid to file, newSeqId=100, maxSeqId=3 2015-04-04 10:48:10,768 DEBUG [RS_CLOSE_META-localhost:51330-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver 2015-04-04 10:48:10,768 DEBUG [RS_CLOSE_META-localhost:51330-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.SimpleRegionObserver$Legacy 2015-04-04 10:48:10,768 DEBUG [RS_CLOSE_META-localhost:51330-0] coprocessor.CoprocessorHost(270): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2015-04-04 10:48:10,768 INFO [RS_CLOSE_META-localhost:51330-0] regionserver.HRegion(1471): Closed hbase:meta,,1.1588230740 2015-04-04 10:48:10,768 DEBUG [RS_CLOSE_META-localhost:51330-0] handler.CloseRegionHandler(164): Closed hbase:meta,,1.1588230740 2015-04-04 10:48:10,830 INFO [RS_OPEN_META-localhost:51330-0-MetaLogRoller] regionserver.LogRoller(163): LogRoller exiting. 2015-04-04 10:48:10,830 INFO [regionserver/localhost/127.0.0.1:0.leaseChecker] regionserver.Leases(146): regionserver/localhost/127.0.0.1:0.leaseChecker closing leases 2015-04-04 10:48:10,830 INFO [regionserver/localhost/127.0.0.1:0.leaseChecker] regionserver.Leases(149): regionserver/localhost/127.0.0.1:0.leaseChecker closed leases 2015-04-04 10:48:10,830 INFO [regionserver/localhost/127.0.0.1:0.logRoller] regionserver.LogRoller(163): LogRoller exiting. 2015-04-04 10:48:10,928 INFO [RS:0;localhost:51330] regionserver.HRegionServer(1023): stopping server localhost,51330,1428169644428; all regions closed. 2015-04-04 10:48:10,928 DEBUG [RS:0;localhost:51330] wal.FSHLog(1148): Closing WAL writer in /user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/WALs/localhost,51330,1428169644428 2015-04-04 10:48:10,934 INFO [IPC Server handler 6 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741831_1007{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-972b2b96-787f-4599-a3c7-25c2e0281ebc:NORMAL|RBW]]} size 15734 2015-04-04 10:48:11,290 INFO [localhost,51327,1428169644129_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2015-04-04 10:48:11,344 DEBUG [RS:0;localhost:51330] wal.FSHLog(1106): Moved 1 WAL file(s) to /user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/oldWALs 2015-04-04 10:48:11,344 INFO [RS:0;localhost:51330] wal.FSHLog(1109): Closed WAL: FSHLog localhost%2C51330%2C1428169644428..meta:.meta(num 1428169646485) 2015-04-04 10:48:11,345 DEBUG [RS:0;localhost:51330] wal.FSHLog(1148): Closing WAL writer in /user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/WALs/localhost,51330,1428169644428 2015-04-04 10:48:11,348 INFO [IPC Server handler 8 on 51319] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:51320 is added to blk_1073741830_1006{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baa7bc61-a942-48c9-a773-bb5a26e14f03:NORMAL|RBW]]} size 21090 2015-04-04 10:48:11,758 DEBUG [RS:0;localhost:51330] wal.FSHLog(1106): Moved 1 WAL file(s) to /user/apurtell/test-data/39c58a7d-b44c-446b-b613-b6201cdf08c7/oldWALs 2015-04-04 10:48:11,758 INFO [RS:0;localhost:51330] wal.FSHLog(1109): Closed WAL: FSHLog localhost%2C51330%2C1428169644428.default:(num 1428169645616) 2015-04-04 10:48:11,758 DEBUG [RS:0;localhost:51330] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:48:11,760 DEBUG [RpcServer.reader=1,bindAddress=localhost,port=51327] ipc.RpcServer$Listener(861): RpcServer.listener,port=51327: DISCONNECTING client 127.0.0.1:51343 because read count=-1. Number of active connections: 1 2015-04-04 10:48:11,765 INFO [M:0;localhost:51327] master.ServerManager(546): Waiting on regionserver(s) to go down localhost,51330,1428169644428 2015-04-04 10:48:11,862 INFO [RS:0;localhost:51330] regionserver.Leases(146): RS:0;localhost:51330 closing leases 2015-04-04 10:48:11,863 INFO [RS:0;localhost:51330] regionserver.Leases(149): RS:0;localhost:51330 closed leases 2015-04-04 10:48:11,863 INFO [RS:0;localhost:51330] hbase.ChoreService(303): Chore service for: localhost,51330,1428169644428 had [java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@574e0984] on shutdown 2015-04-04 10:48:11,874 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/localhost,51330,1428169644428 2015-04-04 10:48:11,874 INFO [RS:0;localhost:51330] client.ConnectionManager$HConnectionImplementation(1675): Closing zookeeper sessionid=0x14c858b55e60006 2015-04-04 10:48:11,876 DEBUG [RS:0;localhost:51330] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:48:11,876 INFO [RS:0;localhost:51330] ipc.RpcServer(2193): Stopping server on 51330 2015-04-04 10:48:11,877 INFO [RpcServer.listener,port=51330] ipc.RpcServer$Listener(765): RpcServer.listener,port=51330: stopping 2015-04-04 10:48:11,878 INFO [RpcServer.responder] ipc.RpcServer$Responder(1007): RpcServer.responder: stopped 2015-04-04 10:48:11,878 INFO [RpcServer.responder] ipc.RpcServer$Responder(910): RpcServer.responder: stopping 2015-04-04 10:48:11,879 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/localhost,51330,1428169644428 2015-04-04 10:48:11,879 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/localhost,51330,1428169644428 2015-04-04 10:48:11,879 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): regionserver:51330-0x14c858b55e60001, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-04-04 10:48:11,880 INFO [main-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [localhost,51330,1428169644428] 2015-04-04 10:48:11,880 INFO [main-EventThread] master.ServerManager(611): Cluster shutdown set; localhost,51330,1428169644428 expired; onlineServers=0 2015-04-04 10:48:11,880 DEBUG [M:0;localhost:51327] master.HMaster(1061): Stopping service threads 2015-04-04 10:48:11,880 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2015-04-04 10:48:11,881 INFO [RS:0;localhost:51330] regionserver.HRegionServer(1067): stopping server localhost,51330,1428169644428; zookeeper connection closed. 2015-04-04 10:48:11,881 INFO [RS:0;localhost:51330] regionserver.HRegionServer(1070): RS:0;localhost:51330 exiting 2015-04-04 10:48:11,881 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1dff92c6] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(191): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1dff92c6 2015-04-04 10:48:11,882 INFO [main] util.JVMClusterUtil(317): Shutdown of 1 master(s) and 4 regionserver(s) complete 2015-04-04 10:48:11,882 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(401): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2015-04-04 10:48:11,882 INFO [M:0;localhost:51327] client.ConnectionManager$HConnectionImplementation(1675): Closing zookeeper sessionid=0x14c858b55e60004 2015-04-04 10:48:11,883 DEBUG [main-EventThread] zookeeper.ZKUtil(492): master:51327-0x14c858b55e60000, quorum=localhost:64763, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2015-04-04 10:48:11,883 DEBUG [M:0;localhost:51327] ipc.RpcClientImpl(1105): Stopping rpc client 2015-04-04 10:48:11,985 INFO [M:0;localhost:51327] hbase.ChoreService(303): Chore service for: localhost,51327,1428169644129_splitLogManager_ had [] on shutdown 2015-04-04 10:48:11,985 INFO [M:0;localhost:51327] flush.MasterFlushTableProcedureManager(79): stop: server shutting down. 2015-04-04 10:48:11,986 INFO [M:0;localhost:51327] ipc.RpcServer(2193): Stopping server on 51327 2015-04-04 10:48:11,986 INFO [RpcServer.listener,port=51327] ipc.RpcServer$Listener(765): RpcServer.listener,port=51327: stopping 2015-04-04 10:48:11,987 INFO [RpcServer.responder] ipc.RpcServer$Responder(1007): RpcServer.responder: stopped 2015-04-04 10:48:11,987 INFO [RpcServer.responder] ipc.RpcServer$Responder(910): RpcServer.responder: stopping 2015-04-04 10:48:11,997 INFO [M:0;localhost:51327] zookeeper.RecoverableZooKeeper(189): Node /hbase/rs/localhost,51327,1428169644129 already deleted, retry=false 2015-04-04 10:48:11,999 INFO [M:0;localhost:51327] regionserver.HRegionServer(1067): stopping server localhost,51327,1428169644129; zookeeper connection closed. 2015-04-04 10:48:11,999 INFO [M:0;localhost:51327] regionserver.HRegionServer(1070): M:0;localhost:51327 exiting 2015-04-04 10:48:12,003 INFO [main] zookeeper.MiniZooKeeperCluster(319): Shutdown MiniZK cluster with all ZK servers 2015-04-04 10:48:12,004 WARN [main] datanode.DirectoryScanner(375): DirectoryScanner: shutdown has been called 2015-04-04 10:48:12,006 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2015-04-04 10:48:12,010 WARN [DataNode: [[[DISK]file:/Users/apurtell/src/hbase/hbase-server/target/test-data/9ec19080-23b1-4378-ad76-121be0b02210/dfscluster_dfa4c3af-a72f-4bda-907a-8271caa6cf61/dfs/data/data1/, [DISK]file:/Users/apurtell/src/hbase/hbase-server/target/test-data/9ec19080-23b1-4378-ad76-121be0b02210/dfscluster_dfa4c3af-a72f-4bda-907a-8271caa6cf61/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:51319] datanode.BPServiceActor(722): BPOfferService for Block pool BP-1536763570-127.0.0.1-1428169641446 (Datanode Uuid 8331f1b7-5f64-4c05-82b9-c129b7261397) service to localhost/127.0.0.1:51319 interrupted 2015-04-04 10:48:12,010 WARN [DataNode: [[[DISK]file:/Users/apurtell/src/hbase/hbase-server/target/test-data/9ec19080-23b1-4378-ad76-121be0b02210/dfscluster_dfa4c3af-a72f-4bda-907a-8271caa6cf61/dfs/data/data1/, [DISK]file:/Users/apurtell/src/hbase/hbase-server/target/test-data/9ec19080-23b1-4378-ad76-121be0b02210/dfscluster_dfa4c3af-a72f-4bda-907a-8271caa6cf61/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:51319] datanode.BPServiceActor(845): Ending block pool service for: Block pool BP-1536763570-127.0.0.1-1428169641446 (Datanode Uuid 8331f1b7-5f64-4c05-82b9-c129b7261397) service to localhost/127.0.0.1:51319 2015-04-04 10:48:12,021 WARN [org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor@1ddaf92] blockmanagement.DecommissionManager$Monitor(78): Monitor interrupted: java.lang.InterruptedException: sleep interrupted 2015-04-04 10:48:12,029 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2015-04-04 10:48:12,046 INFO [main] hbase.HBaseTestingUtility(1062): Minicluster is down Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.724 sec - in org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 2015-04-04 10:48:12,050 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@39a3014f 2015-04-04 10:48:12,050 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-04-04 10:48:12,050 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@39a3014f 2015-04-04 10:48:12,050 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-04-04 10:48:12,050 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@39a3014f 2015-04-04 10:48:12,050 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2015-04-04 10:48:12,050 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@39a3014f 2015-04-04 10:48:12,050 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(120): Starting fs shutdown hook thread. 2015-04-04 10:48:12,051 INFO [Thread-1] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. Results : Tests run: 13, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] --- maven-surefire-plugin:2.18:test (secondPartTestsExecution) @ hbase-server --- [INFO] Tests are skipped. [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building HBase - Testing Util 1.1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-enforcer-plugin:1.0.1:enforce (enforce) @ hbase-testing-util --- [INFO] [INFO] --- jacoco-maven-plugin:0.6.2.201302030002:prepare-agent (prepare-agent) @ hbase-testing-util --- [INFO] Skipping JaCoCo execution [INFO] argLine set to [INFO] [INFO] --- maven-remote-resources-plugin:1.4:process (default) @ hbase-testing-util --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hbase-testing-util --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/apurtell/src/hbase/hbase-testing-util/src/main/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hbase-testing-util --- [INFO] No sources to compile [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hbase-testing-util --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/apurtell/src/hbase/hbase-testing-util/src/test/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hbase-testing-util --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.18:test (default-test) @ hbase-testing-util --- [INFO] [INFO] --- maven-surefire-plugin:2.18:test (secondPartTestsExecution) @ hbase-testing-util --- [INFO] Tests are skipped. [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building HBase - Thrift 1.1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-enforcer-plugin:1.0.1:enforce (enforce) @ hbase-thrift --- [INFO] [INFO] --- jacoco-maven-plugin:0.6.2.201302030002:prepare-agent (prepare-agent) @ hbase-thrift --- [INFO] Skipping JaCoCo execution [INFO] argLine set to [INFO] [INFO] --- maven-antrun-plugin:1.6:run (generate) @ hbase-thrift --- [INFO] Executing tasks main: [mkdir] Created dir: /Users/apurtell/src/hbase/hbase-thrift/target/hbase-webapps [copy] Copying 17 files to /Users/apurtell/src/hbase/hbase-thrift/target/hbase-webapps [INFO] Logging to org.slf4j.impl.SimpleLogger(org.mortbay.log) via org.mortbay.log.Slf4jLog [mkdir] Created dir: /Users/apurtell/src/hbase/hbase-thrift/target/hbase-webapps/thrift/WEB-INF [INFO] Executed tasks [INFO] [INFO] --- build-helper-maven-plugin:1.9.1:add-source (jspcSource-packageInfo-source) @ hbase-thrift --- [INFO] Source directory: /Users/apurtell/src/hbase/hbase-thrift/target/generated-sources/java added. [INFO] [INFO] --- maven-remote-resources-plugin:1.4:process (default) @ hbase-thrift --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hbase-thrift --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 18 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hbase-thrift --- [INFO] Compiling 50 source files to /Users/apurtell/src/hbase/hbase-thrift/target/classes [INFO] [INFO] --- maven-dependency-plugin:2.8:build-classpath (create-mrapp-generated-classpath) @ hbase-thrift --- [INFO] Wrote classpath file '/Users/apurtell/src/hbase/hbase-thrift/target/test-classes/mrapp-generated-classpath'. [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hbase-thrift --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/apurtell/src/hbase/hbase-thrift/src/test/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hbase-thrift --- [INFO] Compiling 6 source files to /Users/apurtell/src/hbase/hbase-thrift/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.18:test (default-test) @ hbase-thrift --- [INFO] [INFO] --- maven-surefire-plugin:2.18:test (secondPartTestsExecution) @ hbase-thrift --- [INFO] Tests are skipped. [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building HBase - Rest 1.1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-enforcer-plugin:1.0.1:enforce (enforce) @ hbase-rest --- [INFO] [INFO] --- jacoco-maven-plugin:0.6.2.201302030002:prepare-agent (prepare-agent) @ hbase-rest --- [INFO] Skipping JaCoCo execution [INFO] argLine set to [INFO] [INFO] --- maven-antrun-plugin:1.6:run (generate) @ hbase-rest --- [INFO] Executing tasks main: [mkdir] Created dir: /Users/apurtell/src/hbase/hbase-rest/target/hbase-webapps [copy] Copying 1 file to /Users/apurtell/src/hbase/hbase-rest/target/hbase-webapps [INFO] Logging to org.slf4j.impl.SimpleLogger(org.mortbay.log) via org.mortbay.log.Slf4jLog [mkdir] Created dir: /Users/apurtell/src/hbase/hbase-rest/target/hbase-webapps/rest/WEB-INF [INFO] Executed tasks [INFO] [INFO] --- build-helper-maven-plugin:1.9.1:add-source (jspcSource-packageInfo-source) @ hbase-rest --- [INFO] Source directory: /Users/apurtell/src/hbase/hbase-rest/target/generated-sources/java added. [INFO] [INFO] --- maven-remote-resources-plugin:1.4:process (default) @ hbase-rest --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hbase-rest --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hbase-rest --- [INFO] Compiling 65 source files to /Users/apurtell/src/hbase/hbase-rest/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hbase-rest --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 6 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hbase-rest --- [INFO] Compiling 34 source files to /Users/apurtell/src/hbase/hbase-rest/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.18:test (default-test) @ hbase-rest --- [INFO] [INFO] --- maven-surefire-plugin:2.18:test (secondPartTestsExecution) @ hbase-rest --- [INFO] Tests are skipped. [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building HBase - Shell 1.1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-enforcer-plugin:1.0.1:enforce (enforce) @ hbase-shell --- [INFO] [INFO] --- jacoco-maven-plugin:0.6.2.201302030002:prepare-agent (prepare-agent) @ hbase-shell --- [INFO] Skipping JaCoCo execution [INFO] argLine set to [INFO] [INFO] --- build-helper-maven-plugin:1.9.1:add-source (jspcSource-packageInfo-source) @ hbase-shell --- [INFO] Source directory: /Users/apurtell/src/hbase/hbase-shell/target/generated-jamon added. [INFO] Source directory: /Users/apurtell/src/hbase/hbase-shell/target/generated-sources/java added. [INFO] [INFO] --- maven-remote-resources-plugin:1.4:process (default) @ hbase-shell --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hbase-shell --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 0 resource [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hbase-shell --- [INFO] No sources to compile [INFO] [INFO] --- maven-dependency-plugin:2.8:build-classpath (create-mrapp-generated-classpath) @ hbase-shell --- [INFO] Wrote classpath file '/Users/apurtell/src/hbase/hbase-shell/target/test-classes/mrapp-generated-classpath'. [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hbase-shell --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/apurtell/src/hbase/hbase-shell/src/test/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hbase-shell --- [INFO] Compiling 3 source files to /Users/apurtell/src/hbase/hbase-shell/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.18:test (default-test) @ hbase-shell --- [INFO] [INFO] --- maven-surefire-plugin:2.18:test (secondPartTestsExecution) @ hbase-shell --- [INFO] Tests are skipped. [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building HBase - Integration Tests 1.1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-enforcer-plugin:1.0.1:enforce (enforce) @ hbase-it --- [INFO] [INFO] --- jacoco-maven-plugin:0.6.2.201302030002:prepare-agent (prepare-agent) @ hbase-it --- [INFO] Skipping JaCoCo execution [INFO] argLine set to [INFO] [INFO] --- maven-remote-resources-plugin:1.4:process (default) @ hbase-it --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hbase-it --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/apurtell/src/hbase/hbase-it/src/main/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hbase-it --- [INFO] No sources to compile [INFO] [INFO] --- maven-dependency-plugin:2.8:build-classpath (create-mrapp-generated-classpath) @ hbase-it --- [INFO] Wrote classpath file '/Users/apurtell/src/hbase/hbase-it/target/test-classes/mrapp-generated-classpath'. [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hbase-it --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 12 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hbase-it --- [INFO] Compiling 79 source files to /Users/apurtell/src/hbase/hbase-it/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.18:test (default-test) @ hbase-it --- [INFO] [INFO] --- maven-surefire-plugin:2.18:test (secondPartTestsExecution) @ hbase-it --- [INFO] Tests are skipped. [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building HBase - Examples 1.1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-enforcer-plugin:1.0.1:enforce (enforce) @ hbase-examples --- [INFO] [INFO] --- jacoco-maven-plugin:0.6.2.201302030002:prepare-agent (prepare-agent) @ hbase-examples --- [INFO] Skipping JaCoCo execution [INFO] argLine set to [INFO] [INFO] --- maven-remote-resources-plugin:1.4:process (default) @ hbase-examples --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hbase-examples --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/apurtell/src/hbase/hbase-examples/src/main/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hbase-examples --- [INFO] Compiling 12 source files to /Users/apurtell/src/hbase/hbase-examples/target/classes [INFO] [INFO] --- maven-dependency-plugin:2.8:build-classpath (create-mrapp-generated-classpath) @ hbase-examples --- [INFO] Wrote classpath file '/Users/apurtell/src/hbase/hbase-examples/target/test-classes/mrapp-generated-classpath'. [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hbase-examples --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/apurtell/src/hbase/hbase-examples/src/test/resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hbase-examples --- [INFO] Compiling 5 source files to /Users/apurtell/src/hbase/hbase-examples/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.18:test (default-test) @ hbase-examples --- [INFO] [INFO] --- maven-surefire-plugin:2.18:test (secondPartTestsExecution) @ hbase-examples --- [INFO] Tests are skipped. [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building HBase - Assembly 1.1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-enforcer-plugin:1.0.1:enforce (enforce) @ hbase-assembly --- [INFO] [INFO] --- jacoco-maven-plugin:0.6.2.201302030002:prepare-agent (prepare-agent) @ hbase-assembly --- [INFO] Skipping JaCoCo for project with packaging type 'pom' [INFO] argLine set to [INFO] [INFO] --- maven-remote-resources-plugin:1.4:process (default) @ hbase-assembly --- [INFO] [INFO] --- maven-dependency-plugin:2.8:build-classpath (create-hbase-generated-classpath) @ hbase-assembly --- [INFO] Wrote classpath file '/Users/apurtell/src/hbase/target/cached_classpath.txt'. [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] HBase .............................................. SUCCESS [ 2.305 s] [INFO] HBase - Checkstyle ................................. SUCCESS [ 0.490 s] [INFO] HBase - Annotations ................................ SUCCESS [ 1.107 s] [INFO] HBase - Common ..................................... SUCCESS [ 9.643 s] [INFO] HBase - Protocol ................................... SUCCESS [ 7.315 s] [INFO] HBase - Client ..................................... SUCCESS [ 3.453 s] [INFO] HBase - Hadoop Compatibility ....................... SUCCESS [ 0.140 s] [INFO] HBase - Hadoop Two Compatibility ................... SUCCESS [ 1.350 s] [INFO] HBase - Prefix Tree ................................ SUCCESS [ 1.111 s] [INFO] HBase - Server ..................................... SUCCESS [01:07 min] [INFO] HBase - Testing Util ............................... SUCCESS [ 1.873 s] [INFO] HBase - Thrift ..................................... SUCCESS [ 2.762 s] [INFO] HBase - Rest ....................................... SUCCESS [ 1.652 s] [INFO] HBase - Shell ...................................... SUCCESS [ 0.716 s] [INFO] HBase - Integration Tests .......................... SUCCESS [ 1.099 s] [INFO] HBase - Examples ................................... SUCCESS [ 1.090 s] [INFO] HBase - Assembly ................................... SUCCESS [ 1.516 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:46 min [INFO] Finished at: 2015-04-04T10:48:22-07:00 [INFO] Final Memory: 115M/838M [INFO] ------------------------------------------------------------------------