2016-10-20 14:32:49,026 INFO [main] hbase.HBaseTestingUtility(497): Created new mini-cluster data directory: /mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/dfscluster_6b2a65f6-e6e0-418a-b6b5-3f3b53b4377b, deleteOnExit=true 2016-10-20 14:32:49,181 INFO [main] zookeeper.MiniZooKeeperCluster(276): Started MiniZooKeeperCluster and ran successful 'stat' on client port=56750 2016-10-20 14:32:49,203 INFO [main] hbase.HBaseTestingUtility(1014): Starting up minicluster with 1 master(s) and 1 regionserver(s) and 1 datanode(s) 2016-10-20 14:32:49,203 INFO [main] hbase.HBaseTestingUtility(744): Setting test.cache.data to /mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/cache_data in system properties and HBase conf 2016-10-20 14:32:49,204 INFO [main] hbase.HBaseTestingUtility(744): Setting hadoop.tmp.dir to /mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/hadoop_tmp in system properties and HBase conf 2016-10-20 14:32:49,204 INFO [main] hbase.HBaseTestingUtility(744): Setting hadoop.log.dir to /mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/hadoop_logs in system properties and HBase conf 2016-10-20 14:32:49,205 INFO [main] hbase.HBaseTestingUtility(744): Setting mapreduce.cluster.local.dir to /mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/mapred_local in system properties and HBase conf 2016-10-20 14:32:49,205 INFO [main] hbase.HBaseTestingUtility(744): Setting mapreduce.cluster.temp.dir to /mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/mapred_temp in system properties and HBase conf 2016-10-20 14:32:49,206 INFO [main] hbase.HBaseTestingUtility(735): read short circuit is OFF 2016-10-20 14:32:49,307 WARN [main] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2016-10-20 14:32:49,482 DEBUG [main] fs.HFileSystem(221): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2016-10-20 14:32:50,662 WARN [main] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2016-10-20 14:32:50,799 INFO [main] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2016-10-20 14:32:50,857 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2016-10-20 14:32:50,889 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/hbase/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.1/hadoop-hdfs-2.7.1-tests.jar!/webapps/hdfs to /tmp/Jetty_localhost_42347_hdfs____ljrb6q/webapp 2016-10-20 14:32:51,059 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42347 2016-10-20 14:32:51,709 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2016-10-20 14:32:51,713 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/hbase/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.1/hadoop-hdfs-2.7.1-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_35709_datanode____8d4ukl/webapp 2016-10-20 14:32:51,833 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35709 2016-10-20 14:32:53,271 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(1862): BLOCK* processReport: from storage DS-2ed22b33-45b0-4f46-8038-08522ec2da53 node DatanodeRegistration(127.0.0.1:38642, datanodeUuid=1e9f9ed6-67b6-46b4-90c6-7f57e5b27722, infoPort=32786, infoSecurePort=0, ipcPort=39097, storageInfo=lv=-56;cid=testClusterID;nsid=1907991453;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs 2016-10-20 14:32:53,271 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(1862): BLOCK* processReport: from storage DS-c31187f8-167e-4840-a137-2273c32903cb node DatanodeRegistration(127.0.0.1:38642, datanodeUuid=1e9f9ed6-67b6-46b4-90c6-7f57e5b27722, infoPort=32786, infoSecurePort=0, ipcPort=39097, storageInfo=lv=-56;cid=testClusterID;nsid=1907991453;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2016-10-20 14:32:53,373 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-20 14:32:53,378 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-20 14:32:53,781 INFO [IPC Server handler 4 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 7 2016-10-20 14:32:54,200 INFO [main] util.FSUtils(749): Created version file at hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e with version=8 2016-10-20 14:32:54,280 DEBUG [main] impl.BackupManager(160): Added region procedure manager: org.apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureManager 2016-10-20 14:32:54,423 INFO [main] client.ConnectionUtils(106): master/cn012.l42scl.hortonworks.com/172.18.128.12:0 server-side HConnection retries=350 2016-10-20 14:32:54,434 INFO [main] ipc.SimpleRpcScheduler(190): Using deadline as user call queue, count=1 2016-10-20 14:32:54,443 INFO [main] ipc.RpcServer$Listener(635): master/cn012.l42scl.hortonworks.com/172.18.128.12:0: started 3 reader(s) listening on port=41289 2016-10-20 14:32:54,478 INFO [main] hfile.CacheConfig(548): Allocating LruBlockCache size=995.60 MB, blockSize=64 KB 2016-10-20 14:32:54,482 DEBUG [main] hfile.CacheConfig(562): Trying to use Internal l2 cache 2016-10-20 14:32:54,483 INFO [main] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:32:54,483 INFO [main] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:32:54,486 INFO [main] mob.MobFileCache(121): MobFileCache is initialized, and the cache size is 1000 2016-10-20 14:32:54,488 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-20 14:32:54,534 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=master:41289 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:32:54,634 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:412890x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:32:54,638 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): master:41289-0x157e2823ec60000 connected 2016-10-20 14:32:54,737 DEBUG [main] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/master 2016-10-20 14:32:54,738 DEBUG [main] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/running 2016-10-20 14:32:54,741 INFO [RpcServer.responder] ipc.RpcServer$Responder(958): RpcServer.responder: starting 2016-10-20 14:32:54,742 INFO [RpcServer.listener,port=41289] ipc.RpcServer$Listener(769): RpcServer.listener,port=41289: starting 2016-10-20 14:32:54,743 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=0 queue=0 2016-10-20 14:32:54,744 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=1 queue=0 2016-10-20 14:32:54,744 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=2 queue=0 2016-10-20 14:32:54,745 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=3 queue=0 2016-10-20 14:32:54,745 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=4 queue=0 2016-10-20 14:32:54,746 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=0 queue=0 2016-10-20 14:32:54,747 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=1 queue=1 2016-10-20 14:32:54,747 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=2 queue=0 2016-10-20 14:32:54,748 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=3 queue=1 2016-10-20 14:32:54,748 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=4 queue=0 2016-10-20 14:32:54,748 DEBUG [main] ipc.RpcExecutor(118): Replication Start Handler index=0 queue=0 2016-10-20 14:32:54,749 DEBUG [main] ipc.RpcExecutor(118): Replication Start Handler index=1 queue=0 2016-10-20 14:32:54,749 DEBUG [main] ipc.RpcExecutor(118): Replication Start Handler index=2 queue=0 2016-10-20 14:32:54,760 INFO [main] master.HMaster(401): hbase.rootdir=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e, hbase.cluster.distributed=false 2016-10-20 14:32:54,768 DEBUG [main] impl.BackupManager(136): Added log cleaner: org.apache.hadoop.hbase.backup.master.BackupLogCleaner 2016-10-20 14:32:54,768 DEBUG [main] impl.BackupManager(137): Added master procedure manager: org.apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager 2016-10-20 14:32:54,768 DEBUG [main] impl.BackupManager(138): Added master observer: org.apache.hadoop.hbase.backup.master.BackupController 2016-10-20 14:32:54,778 INFO [main] master.HMaster(1725): Adding backup master ZNode /1/backup-masters/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:54,790 DEBUG [main] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/backup-masters/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:54,806 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/master 2016-10-20 14:32:54,807 DEBUG [cn012:41289.activeMasterManager] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/master 2016-10-20 14:32:54,809 INFO [cn012:41289.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /1/backup-masters/cn012.l42scl.hortonworks.com,41289,1476973974467 from backup master directory 2016-10-20 14:32:54,822 DEBUG [main-EventThread] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/master 2016-10-20 14:32:54,822 DEBUG [main-EventThread] master.ActiveMasterManager(126): A master is now available 2016-10-20 14:32:54,823 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/backup-masters/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:54,823 WARN [cn012:41289.activeMasterManager] hbase.ZNodeClearer(61): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2016-10-20 14:32:54,823 INFO [cn012:41289.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:54,883 DEBUG [main] impl.BackupManager(160): Added region procedure manager: org.apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureManager 2016-10-20 14:32:54,885 INFO [main] client.ConnectionUtils(106): regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0 server-side HConnection retries=350 2016-10-20 14:32:54,885 INFO [main] ipc.SimpleRpcScheduler(190): Using deadline as user call queue, count=1 2016-10-20 14:32:54,886 INFO [main] ipc.RpcServer$Listener(635): regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0: started 3 reader(s) listening on port=37560 2016-10-20 14:32:54,890 INFO [main] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:32:54,890 INFO [main] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:32:54,892 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-20 14:32:54,896 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:37560 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:32:54,905 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:375600x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:32:54,908 DEBUG [main] zookeeper.ZKUtil(365): regionserver:375600x0, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/master 2016-10-20 14:32:54,909 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): regionserver:37560-0x157e2823ec60001 connected 2016-10-20 14:32:54,911 DEBUG [main] zookeeper.ZKUtil(367): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/running 2016-10-20 14:32:54,911 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=0 queue=0 2016-10-20 14:32:54,911 INFO [RpcServer.responder] ipc.RpcServer$Responder(958): RpcServer.responder: starting 2016-10-20 14:32:54,911 INFO [RpcServer.listener,port=37560] ipc.RpcServer$Listener(769): RpcServer.listener,port=37560: starting 2016-10-20 14:32:54,912 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=1 queue=0 2016-10-20 14:32:54,913 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=2 queue=0 2016-10-20 14:32:54,913 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=3 queue=0 2016-10-20 14:32:54,913 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=4 queue=0 2016-10-20 14:32:54,914 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=0 queue=0 2016-10-20 14:32:54,914 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=1 queue=1 2016-10-20 14:32:54,914 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=2 queue=0 2016-10-20 14:32:54,915 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=3 queue=1 2016-10-20 14:32:54,915 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=4 queue=0 2016-10-20 14:32:54,915 DEBUG [main] ipc.RpcExecutor(118): Replication Start Handler index=0 queue=0 2016-10-20 14:32:54,915 DEBUG [main] ipc.RpcExecutor(118): Replication Start Handler index=1 queue=0 2016-10-20 14:32:54,916 DEBUG [main] ipc.RpcExecutor(118): Replication Start Handler index=2 queue=0 2016-10-20 14:32:54,938 INFO [IPC Server handler 4 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:32:54,941 DEBUG [cn012:41289.activeMasterManager] util.FSUtils(901): Created cluster ID file at hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/hbase.id with ID: d6c08ff7-c189-481f-b5e4-7aa260ec3e69 2016-10-20 14:32:54,948 INFO [RS:0;cn012:37560] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x3437f44b connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:32:54,948 INFO [M:0;cn012:41289] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x4227cb7 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:32:54,955 DEBUG [RS:0;cn012:37560-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x3437f44b0x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:32:54,955 DEBUG [M:0;cn012:41289-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x4227cb70x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:32:54,955 INFO [RS:0;cn012:37560] client.ZooKeeperRegistry(104): ClusterId read in ZooKeeper is null 2016-10-20 14:32:54,955 INFO [M:0;cn012:41289] client.ZooKeeperRegistry(104): ClusterId read in ZooKeeper is null 2016-10-20 14:32:54,956 DEBUG [RS:0;cn012:37560] client.ConnectionImplementation(466): clusterid came back null, using default default-cluster 2016-10-20 14:32:54,957 DEBUG [RS:0;cn012:37560-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x3437f44b-0x157e2823ec60002 connected 2016-10-20 14:32:54,956 DEBUG [M:0;cn012:41289-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x4227cb7-0x157e2823ec60003 connected 2016-10-20 14:32:54,956 DEBUG [M:0;cn012:41289] client.ConnectionImplementation(466): clusterid came back null, using default default-cluster 2016-10-20 14:32:54,971 DEBUG [M:0;cn012:41289] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ae58243, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:32:54,971 DEBUG [RS:0;cn012:37560] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@444cf248, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:32:54,971 DEBUG [M:0;cn012:41289] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:32:54,971 DEBUG [RS:0;cn012:37560] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:32:54,972 DEBUG [M:0;cn012:41289] ipc.AsyncRpcClient(138): Create NioEventLoopGroup with maxThreads = 0 2016-10-20 14:32:54,979 DEBUG [M:0;cn012:41289] ipc.AsyncRpcClient(113): Create global event loop group NioEventLoopGroup 2016-10-20 14:32:54,979 DEBUG [M:0;cn012:41289] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:32:54,979 DEBUG [RS:0;cn012:37560] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:32:54,982 INFO [cn012:41289.activeMasterManager] master.MasterFileSystem(528): BOOTSTRAP: creating hbase:meta region 2016-10-20 14:32:54,985 INFO [cn012:41289.activeMasterManager] regionserver.HRegion(6162): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', BLOOMFILTER => 'NONE', VERSIONS => '3', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'false', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e Table name == hbase:meta 2016-10-20 14:32:55,013 INFO [IPC Server handler 5 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:32:55,018 DEBUG [cn012:41289.activeMasterManager] regionserver.HRegion(736): Instantiated hbase:meta,,1.1588230740 2016-10-20 14:32:55,067 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:32:55,075 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:32:55,079 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/info 2016-10-20 14:32:55,086 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:32:55,087 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:32:55,088 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/table 2016-10-20 14:32:55,100 DEBUG [cn012:41289.activeMasterManager] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740 2016-10-20 14:32:55,116 DEBUG [cn012:41289.activeMasterManager] regionserver.FlushLargeStoresPolicy(72): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table hbase:meta, use config (67108864) instead 2016-10-20 14:32:55,124 DEBUG [cn012:41289.activeMasterManager] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:32:55,124 INFO [cn012:41289.activeMasterManager] regionserver.HRegion(871): Onlined 1588230740; next sequenceid=2 2016-10-20 14:32:55,125 DEBUG [cn012:41289.activeMasterManager] regionserver.HRegion(1419): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2016-10-20 14:32:55,125 DEBUG [cn012:41289.activeMasterManager] regionserver.HRegion(1446): Updates disabled for region hbase:meta,,1.1588230740 2016-10-20 14:32:55,126 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(839): Closed info 2016-10-20 14:32:55,126 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(839): Closed table 2016-10-20 14:32:55,126 INFO [cn012:41289.activeMasterManager] regionserver.HRegion(1552): Closed hbase:meta,,1.1588230740 2016-10-20 14:32:55,157 INFO [IPC Server handler 3 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:32:55,162 DEBUG [cn012:41289.activeMasterManager] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2016-10-20 14:32:55,178 INFO [cn012:41289.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-20 14:32:55,185 INFO [cn012:41289.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2016-10-20 14:32:55,186 DEBUG [cn012:41289.activeMasterManager] util.FSTableDescriptors(222): Fetching table descriptors from the filesystem. 2016-10-20 14:32:55,225 INFO [cn012:41289.activeMasterManager] balancer.StochasticLoadBalancer(156): loading config 2016-10-20 14:32:55,233 DEBUG [cn012:41289.activeMasterManager] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/balancer 2016-10-20 14:32:55,234 DEBUG [cn012:41289.activeMasterManager] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/normalizer 2016-10-20 14:32:55,248 DEBUG [cn012:41289.activeMasterManager] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/switch/split 2016-10-20 14:32:55,248 DEBUG [cn012:41289.activeMasterManager] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/switch/merge 2016-10-20 14:32:55,279 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/running 2016-10-20 14:32:55,280 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/running 2016-10-20 14:32:55,280 INFO [cn012:41289.activeMasterManager] master.HMaster(624): Server active/primary master=cn012.l42scl.hortonworks.com,41289,1476973974467, sessionid=0x157e2823ec60000, setting cluster-up flag (Was=false) 2016-10-20 14:32:55,284 INFO [M:0;cn012:41289] regionserver.HRegionServer(813): ClusterId : d6c08ff7-c189-481f-b5e4-7aa260ec3e69 2016-10-20 14:32:55,284 INFO [RS:0;cn012:37560] regionserver.HRegionServer(813): ClusterId : d6c08ff7-c189-481f-b5e4-7aa260ec3e69 2016-10-20 14:32:55,286 INFO [M:0;cn012:41289] procedure.ProcedureManagerHost(71): User procedure org.apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureManager was loaded successfully. 2016-10-20 14:32:55,286 INFO [RS:0;cn012:37560] procedure.ProcedureManagerHost(71): User procedure org.apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureManager was loaded successfully. 2016-10-20 14:32:55,288 DEBUG [M:0;cn012:41289] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc is initializing 2016-10-20 14:32:55,288 DEBUG [RS:0;cn012:37560] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc is initializing 2016-10-20 14:32:55,290 INFO [cn012:41289.activeMasterManager] procedure.ProcedureManagerHost(71): User procedure org.apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager was loaded successfully. 2016-10-20 14:32:55,321 DEBUG [cn012:41289.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /1/flush-table-proc already exists 2016-10-20 14:32:55,321 DEBUG [RS:0;cn012:37560] zookeeper.RecoverableZooKeeper(594): Node /1/flush-table-proc already exists 2016-10-20 14:32:55,338 DEBUG [cn012:41289.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /1/flush-table-proc/acquired already exists 2016-10-20 14:32:55,338 DEBUG [RS:0;cn012:37560] zookeeper.RecoverableZooKeeper(594): Node /1/flush-table-proc/acquired already exists 2016-10-20 14:32:55,355 DEBUG [RS:0;cn012:37560] zookeeper.RecoverableZooKeeper(594): Node /1/flush-table-proc/abort already exists 2016-10-20 14:32:55,356 DEBUG [M:0;cn012:41289] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc is initialized 2016-10-20 14:32:55,356 DEBUG [RS:0;cn012:37560] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc is initialized 2016-10-20 14:32:55,356 DEBUG [M:0;cn012:41289] procedure.RegionServerProcedureManagerHost(43): Procedure backup-proc is initializing 2016-10-20 14:32:55,356 DEBUG [RS:0;cn012:37560] procedure.RegionServerProcedureManagerHost(43): Procedure backup-proc is initializing 2016-10-20 14:32:55,363 DEBUG [cn012:41289.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /1/flush-table-proc/abort already exists 2016-10-20 14:32:55,363 INFO [cn012:41289.activeMasterManager] procedure.ZKProcedureUtil(270): Clearing all procedure znodes: /1/flush-table-proc/acquired /1/flush-table-proc/reached /1/flush-table-proc/abort 2016-10-20 14:32:55,380 DEBUG [M:0;cn012:41289] zookeeper.RecoverableZooKeeper(594): Node /1/rolllog-proc already exists 2016-10-20 14:32:55,388 DEBUG [cn012:41289.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(248): Starting the controller for procedure member:cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:55,396 DEBUG [M:0;cn012:41289] zookeeper.RecoverableZooKeeper(594): Node /1/rolllog-proc/acquired already exists 2016-10-20 14:32:55,405 DEBUG [cn012:41289.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /1/rolllog-proc/acquired already exists 2016-10-20 14:32:55,422 DEBUG [M:0;cn012:41289] procedure.RegionServerProcedureManagerHost(45): Procedure backup-proc is initialized 2016-10-20 14:32:55,422 DEBUG [M:0;cn012:41289] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot is initializing 2016-10-20 14:32:55,430 DEBUG [RS:0;cn012:37560] zookeeper.RecoverableZooKeeper(594): Node /1/rolllog-proc/abort already exists 2016-10-20 14:32:55,430 DEBUG [cn012:41289.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /1/rolllog-proc/abort already exists 2016-10-20 14:32:55,430 DEBUG [RS:0;cn012:37560] procedure.RegionServerProcedureManagerHost(45): Procedure backup-proc is initialized 2016-10-20 14:32:55,431 DEBUG [RS:0;cn012:37560] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot is initializing 2016-10-20 14:32:55,430 INFO [cn012:41289.activeMasterManager] procedure.ZKProcedureUtil(270): Clearing all procedure znodes: /1/rolllog-proc/acquired /1/rolllog-proc/reached /1/rolllog-proc/abort 2016-10-20 14:32:55,463 DEBUG [RS:0;cn012:37560] zookeeper.RecoverableZooKeeper(594): Node /1/online-snapshot already exists 2016-10-20 14:32:55,472 DEBUG [cn012:41289.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(248): Starting the controller for procedure member:cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:55,480 DEBUG [RS:0;cn012:37560] zookeeper.RecoverableZooKeeper(594): Node /1/online-snapshot/acquired already exists 2016-10-20 14:32:55,488 DEBUG [cn012:41289.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /1/online-snapshot/acquired already exists 2016-10-20 14:32:55,530 DEBUG [M:0;cn012:41289] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot is initialized 2016-10-20 14:32:55,535 INFO [M:0;cn012:41289] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2016-10-20 14:32:55,538 DEBUG [cn012:41289.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /1/online-snapshot/abort already exists 2016-10-20 14:32:55,538 DEBUG [RS:0;cn012:37560] zookeeper.RecoverableZooKeeper(594): Node /1/online-snapshot/abort already exists 2016-10-20 14:32:55,538 INFO [cn012:41289.activeMasterManager] procedure.ZKProcedureUtil(270): Clearing all procedure znodes: /1/online-snapshot/acquired /1/online-snapshot/reached /1/online-snapshot/abort 2016-10-20 14:32:55,538 DEBUG [RS:0;cn012:37560] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot is initialized 2016-10-20 14:32:55,539 INFO [RS:0;cn012:37560] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2016-10-20 14:32:55,542 DEBUG [cn012:41289.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(248): Starting the controller for procedure member:cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:55,546 INFO [M:0;cn012:41289] throttle.PressureAwareCompactionThroughputController(132): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2016-10-20 14:32:55,546 INFO [RS:0;cn012:37560] throttle.PressureAwareCompactionThroughputController(132): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2016-10-20 14:32:55,548 INFO [RS:0;cn012:37560] regionserver.HRegionServer$CompactionChecker(1555): CompactionChecker runs every 1sec 2016-10-20 14:32:55,548 INFO [M:0;cn012:41289] regionserver.HRegionServer$CompactionChecker(1555): CompactionChecker runs every 1sec 2016-10-20 14:32:55,553 DEBUG [M:0;cn012:41289] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b80b306, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=cn012.l42scl.hortonworks.com/172.18.128.12:0 2016-10-20 14:32:55,553 DEBUG [RS:0;cn012:37560] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b6a88db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=cn012.l42scl.hortonworks.com/172.18.128.12:0 2016-10-20 14:32:55,553 DEBUG [RS:0;cn012:37560] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:32:55,553 DEBUG [RS:0;cn012:37560] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:32:55,553 DEBUG [M:0;cn012:41289] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:32:55,554 DEBUG [M:0;cn012:41289] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:32:55,558 DEBUG [M:0;cn012:41289] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:M:0;cn012:41289 2016-10-20 14:32:55,558 DEBUG [RS:0;cn012:37560] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:RS:0;cn012:37560 2016-10-20 14:32:55,574 INFO [cn012:41289.activeMasterManager] master.MasterCoprocessorHost(91): System coprocessor loading is enabled 2016-10-20 14:32:55,580 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rs 2016-10-20 14:32:55,580 INFO [cn012:41289.activeMasterManager] coprocessor.CoprocessorHost(161): System coprocessor org.apache.hadoop.hbase.backup.master.BackupController was loaded successfully with priority (536870911). 2016-10-20 14:32:55,582 DEBUG [cn012:41289.activeMasterManager] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-cn012:41289, corePoolSize=5, maxPoolSize=5 2016-10-20 14:32:55,582 DEBUG [cn012:41289.activeMasterManager] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-cn012:41289, corePoolSize=5, maxPoolSize=5 2016-10-20 14:32:55,582 DEBUG [cn012:41289.activeMasterManager] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-cn012:41289, corePoolSize=5, maxPoolSize=5 2016-10-20 14:32:55,583 DEBUG [cn012:41289.activeMasterManager] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-cn012:41289, corePoolSize=5, maxPoolSize=5 2016-10-20 14:32:55,583 DEBUG [cn012:41289.activeMasterManager] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-cn012:41289, corePoolSize=10, maxPoolSize=10 2016-10-20 14:32:55,583 DEBUG [cn012:41289.activeMasterManager] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-cn012:41289, corePoolSize=1, maxPoolSize=1 2016-10-20 14:32:55,588 DEBUG [M:0;cn012:41289] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rs/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:55,588 DEBUG [RS:0;cn012:37560] zookeeper.ZKUtil(365): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rs/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:32:55,589 DEBUG [main-EventThread] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rs/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:32:55,590 DEBUG [main-EventThread] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rs/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:55,592 INFO [M:0;cn012:41289] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2016-10-20 14:32:55,592 INFO [RS:0;cn012:37560] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2016-10-20 14:32:55,593 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /1/rs/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:32:55,593 INFO [M:0;cn012:41289] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2016-10-20 14:32:55,594 INFO [RS:0;cn012:37560] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2016-10-20 14:32:55,594 INFO [M:0;cn012:41289] regionserver.HRegionServer(2344): reportForDuty to master=cn012.l42scl.hortonworks.com,41289,1476973974467 with port=41289, startcode=1476973974467 2016-10-20 14:32:55,595 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /1/rs/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:55,596 INFO [RS:0;cn012:37560] regionserver.HRegionServer(2344): reportForDuty to master=cn012.l42scl.hortonworks.com,41289,1476973974467 with port=37560, startcode=1476973974890 2016-10-20 14:32:55,597 DEBUG [M:0;cn012:41289] regionserver.HRegionServer(2363): Master is not running yet 2016-10-20 14:32:55,597 WARN [M:0;cn012:41289] regionserver.HRegionServer(941): reportForDuty failed; sleeping and then retrying. 2016-10-20 14:32:55,600 INFO [cn012:41289.activeMasterManager] procedure2.ProcedureExecutor(487): Starting procedure executor threads=25 2016-10-20 14:32:55,601 INFO [cn012:41289.activeMasterManager] wal.WALProcedureStore(296): Starting WAL Procedure Store lease recovery 2016-10-20 14:32:55,603 WARN [cn012:41289.activeMasterManager] wal.WALProcedureStore(941): Log directory not found: File hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/MasterProcWALs does not exist. 2016-10-20 14:32:55,610 DEBUG [cn012:41289.activeMasterManager] wal.WALProcedureStore(833): Roll new state log: 1 2016-10-20 14:32:55,612 INFO [cn012:41289.activeMasterManager] wal.WALProcedureStore(319): Lease acquired for flushLogId: 1 2016-10-20 14:32:55,612 DEBUG [cn012:41289.activeMasterManager] wal.WALProcedureStore(336): No state logs to replay. 2016-10-20 14:32:55,612 DEBUG [cn012:41289.activeMasterManager] procedure2.ProcedureExecutor$1(298): load procedures maxProcId=0 2016-10-20 14:32:55,619 DEBUG [cn012:41289.activeMasterManager] cleaner.CleanerChore(99): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2016-10-20 14:32:55,621 INFO [cn012:41289.activeMasterManager] zookeeper.RecoverableZooKeeper(120): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:32:55,630 DEBUG [cn012:41289.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(590): replicationLogCleaner0x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:32:55,632 DEBUG [cn012:41289.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(674): replicationLogCleaner-0x157e2823ec60004 connected 2016-10-20 14:32:55,644 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:50334; # active connections: 1 2016-10-20 14:32:55,656 DEBUG [cn012:41289.activeMasterManager] cleaner.CleanerChore(99): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2016-10-20 14:32:55,657 INFO [cn012:41289.activeMasterManager] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x3a112c11 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:32:55,663 DEBUG [cn012:41289.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x3a112c110x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:32:55,665 DEBUG [cn012:41289.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x3a112c11-0x157e2823ec60005 connected 2016-10-20 14:32:55,665 DEBUG [cn012:41289.activeMasterManager] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@620da698, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:32:55,665 DEBUG [cn012:41289.activeMasterManager] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:32:55,665 DEBUG [cn012:41289.activeMasterManager] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:32:55,666 DEBUG [cn012:41289.activeMasterManager] cleaner.CleanerChore(99): initialize cleaner=org.apache.hadoop.hbase.backup.master.BackupLogCleaner 2016-10-20 14:32:55,667 DEBUG [cn012:41289.activeMasterManager] cleaner.CleanerChore(99): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2016-10-20 14:32:55,669 DEBUG [cn012:41289.activeMasterManager] cleaner.CleanerChore(99): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2016-10-20 14:32:55,670 DEBUG [cn012:41289.activeMasterManager] cleaner.CleanerChore(99): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2016-10-20 14:32:55,670 INFO [cn012:41289.activeMasterManager] master.ServerManager(1008): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2016-10-20 14:32:55,670 INFO [M:0;cn012:41289] regionserver.HRegionServer(2344): reportForDuty to master=cn012.l42scl.hortonworks.com,41289,1476973974467 with port=41289, startcode=1476973974467 2016-10-20 14:32:55,672 DEBUG [AsyncRpcChannel-pool2-t1] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2016-10-20 14:32:55,680 INFO [M:0;cn012:41289] master.ServerManager(456): Registering server=cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:55,684 INFO [M:0;cn012:41289] regionserver.HRegionServer(1390): Config from master: hbase.rootdir=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e 2016-10-20 14:32:55,684 INFO [M:0;cn012:41289] regionserver.HRegionServer(1390): Config from master: fs.defaultFS=hdfs://localhost:37879 2016-10-20 14:32:55,684 INFO [M:0;cn012:41289] regionserver.HRegionServer(1390): Config from master: hbase.master.info.port=-1 2016-10-20 14:32:55,684 WARN [M:0;cn012:41289] hbase.ZNodeClearer(61): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2016-10-20 14:32:55,684 INFO [M:0;cn012:41289] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:32:55,685 DEBUG [M:0;cn012:41289] regionserver.HRegionServer(1654): logdir=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:55,702 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase.hfs.0 (auth:SIMPLE) 2016-10-20 14:32:55,707 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 50334 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:32:55,710 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.ServerManager(456): Registering server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:32:55,713 DEBUG [M:0;cn012:41289] regionserver.Replication(152): ReplicationStatisticsThread 300 2016-10-20 14:32:55,720 INFO [cn012:41289.activeMasterManager] master.ServerManager(1025): Finished waiting for region servers count to settle; checked in 2, slept for 50 ms, expecting minimum of 1, maximum of 1, master is running 2016-10-20 14:32:55,721 INFO [RS:0;cn012:37560] regionserver.HRegionServer(1390): Config from master: hbase.rootdir=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e 2016-10-20 14:32:55,723 INFO [RS:0;cn012:37560] regionserver.HRegionServer(1390): Config from master: fs.defaultFS=hdfs://localhost:37879 2016-10-20 14:32:55,723 INFO [RS:0;cn012:37560] regionserver.HRegionServer(1390): Config from master: hbase.master.info.port=-1 2016-10-20 14:32:55,723 WARN [RS:0;cn012:37560] hbase.ZNodeClearer(61): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2016-10-20 14:32:55,724 INFO [RS:0;cn012:37560] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:32:55,724 DEBUG [RS:0;cn012:37560] regionserver.HRegionServer(1654): logdir=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:32:55,725 DEBUG [cn012:41289.activeMasterManager] zookeeper.ZKUtil(624): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Unable to get data of znode /1/meta-region-server because node does not exist (not an error) 2016-10-20 14:32:55,725 INFO [M:0;cn012:41289] wal.WALFactory(144): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.RegionGroupingProvider 2016-10-20 14:32:55,728 INFO [M:0;cn012:41289] wal.RegionGroupingProvider(107): Instantiating RegionGroupingStrategy of type class org.apache.hadoop.hbase.wal.BoundedGroupingStrategy 2016-10-20 14:32:55,736 INFO [M:0;cn012:41289] regionserver.MetricsRegionServerWrapperImpl(139): Computing regionserver metrics every 5000 milliseconds 2016-10-20 14:32:55,740 DEBUG [RS:0;cn012:37560] regionserver.Replication(152): ReplicationStatisticsThread 300 2016-10-20 14:32:55,741 INFO [RS:0;cn012:37560] wal.WALFactory(144): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.RegionGroupingProvider 2016-10-20 14:32:55,742 INFO [RS:0;cn012:37560] wal.RegionGroupingProvider(107): Instantiating RegionGroupingStrategy of type class org.apache.hadoop.hbase.wal.BoundedGroupingStrategy 2016-10-20 14:32:55,742 INFO [RS:0;cn012:37560] regionserver.MetricsRegionServerWrapperImpl(139): Computing regionserver metrics every 5000 milliseconds 2016-10-20 14:32:55,745 DEBUG [M:0;cn012:41289] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-cn012:41289, corePoolSize=3, maxPoolSize=3 2016-10-20 14:32:55,746 DEBUG [M:0;cn012:41289] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-cn012:41289, corePoolSize=1, maxPoolSize=1 2016-10-20 14:32:55,747 DEBUG [M:0;cn012:41289] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-cn012:41289, corePoolSize=3, maxPoolSize=3 2016-10-20 14:32:55,751 DEBUG [M:0;cn012:41289] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-cn012:41289, corePoolSize=1, maxPoolSize=1 2016-10-20 14:32:55,753 DEBUG [M:0;cn012:41289] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-cn012:41289, corePoolSize=2, maxPoolSize=2 2016-10-20 14:32:55,754 DEBUG [M:0;cn012:41289] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-cn012:41289, corePoolSize=10, maxPoolSize=10 2016-10-20 14:32:55,754 DEBUG [M:0;cn012:41289] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-cn012:41289, corePoolSize=3, maxPoolSize=3 2016-10-20 14:32:55,757 DEBUG [M:0;cn012:41289] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rs/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:32:55,757 DEBUG [RS:0;cn012:37560] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-cn012:37560, corePoolSize=3, maxPoolSize=3 2016-10-20 14:32:55,758 DEBUG [M:0;cn012:41289] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rs/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:55,758 DEBUG [RS:0;cn012:37560] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-cn012:37560, corePoolSize=1, maxPoolSize=1 2016-10-20 14:32:55,758 INFO [M:0;cn012:41289] regionserver.ReplicationSourceManager(246): Current list of replicators: [cn012.l42scl.hortonworks.com,37560,1476973974890, cn012.l42scl.hortonworks.com,41289,1476973974467] other RSs: [cn012.l42scl.hortonworks.com,37560,1476973974890, cn012.l42scl.hortonworks.com,41289,1476973974467] 2016-10-20 14:32:55,758 DEBUG [RS:0;cn012:37560] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-cn012:37560, corePoolSize=3, maxPoolSize=3 2016-10-20 14:32:55,759 DEBUG [RS:0;cn012:37560] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-cn012:37560, corePoolSize=1, maxPoolSize=1 2016-10-20 14:32:55,759 DEBUG [RS:0;cn012:37560] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-cn012:37560, corePoolSize=2, maxPoolSize=2 2016-10-20 14:32:55,759 DEBUG [RS:0;cn012:37560] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-cn012:37560, corePoolSize=10, maxPoolSize=10 2016-10-20 14:32:55,759 DEBUG [RS:0;cn012:37560] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-cn012:37560, corePoolSize=3, maxPoolSize=3 2016-10-20 14:32:55,761 DEBUG [RS:0;cn012:37560] zookeeper.ZKUtil(365): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rs/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:32:55,762 DEBUG [RS:0;cn012:37560] zookeeper.ZKUtil(365): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rs/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:55,762 INFO [RS:0;cn012:37560] regionserver.ReplicationSourceManager(246): Current list of replicators: [cn012.l42scl.hortonworks.com,37560,1476973974890, cn012.l42scl.hortonworks.com,41289,1476973974467] other RSs: [cn012.l42scl.hortonworks.com,37560,1476973974890, cn012.l42scl.hortonworks.com,41289,1476973974467] 2016-10-20 14:32:55,837 INFO [SplitLogWorker-cn012:41289] regionserver.SplitLogWorker(134): SplitLogWorker cn012.l42scl.hortonworks.com,41289,1476973974467 starting 2016-10-20 14:32:55,837 INFO [SplitLogWorker-cn012:37560] regionserver.SplitLogWorker(134): SplitLogWorker cn012.l42scl.hortonworks.com,37560,1476973974890 starting 2016-10-20 14:32:55,839 INFO [M:0;cn012:41289] regionserver.HeapMemoryManager(191): Starting HeapMemoryTuner chore. 2016-10-20 14:32:55,839 INFO [RS:0;cn012:37560] regionserver.HeapMemoryManager(191): Starting HeapMemoryTuner chore. 2016-10-20 14:32:55,843 INFO [M:0;cn012:41289] regionserver.HRegionServer(1412): Serving as cn012.l42scl.hortonworks.com,41289,1476973974467, RpcServer on cn012.l42scl.hortonworks.com/172.18.128.12:41289, sessionid=0x157e2823ec60000 2016-10-20 14:32:55,843 INFO [RS:0;cn012:37560] regionserver.HRegionServer(1412): Serving as cn012.l42scl.hortonworks.com,37560,1476973974890, RpcServer on cn012.l42scl.hortonworks.com/172.18.128.12:37560, sessionid=0x157e2823ec60001 2016-10-20 14:32:55,843 DEBUG [M:0;cn012:41289] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc is starting 2016-10-20 14:32:55,843 DEBUG [M:0;cn012:41289] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:55,843 DEBUG [M:0;cn012:41289] procedure.ZKProcedureMemberRpcs(356): Starting procedure member 'cn012.l42scl.hortonworks.com,41289,1476973974467' 2016-10-20 14:32:55,843 DEBUG [RS:0;cn012:37560] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc is starting 2016-10-20 14:32:55,843 DEBUG [M:0;cn012:41289] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/flush-table-proc/abort' 2016-10-20 14:32:55,843 DEBUG [RS:0;cn012:37560] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:32:55,844 DEBUG [RS:0;cn012:37560] procedure.ZKProcedureMemberRpcs(356): Starting procedure member 'cn012.l42scl.hortonworks.com,37560,1476973974890' 2016-10-20 14:32:55,844 DEBUG [RS:0;cn012:37560] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/flush-table-proc/abort' 2016-10-20 14:32:55,845 DEBUG [M:0;cn012:41289] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/flush-table-proc/acquired' 2016-10-20 14:32:55,845 DEBUG [RS:0;cn012:37560] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/flush-table-proc/acquired' 2016-10-20 14:32:55,845 DEBUG [M:0;cn012:41289] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc is started 2016-10-20 14:32:55,845 DEBUG [M:0;cn012:41289] procedure.RegionServerProcedureManagerHost(51): Procedure backup-proc is starting 2016-10-20 14:32:55,845 DEBUG [RS:0;cn012:37560] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc is started 2016-10-20 14:32:55,845 DEBUG [M:0;cn012:41289] procedure.ZKProcedureMemberRpcs(356): Starting procedure member 'cn012.l42scl.hortonworks.com,41289,1476973974467' 2016-10-20 14:32:55,845 DEBUG [RS:0;cn012:37560] procedure.RegionServerProcedureManagerHost(51): Procedure backup-proc is starting 2016-10-20 14:32:55,846 DEBUG [M:0;cn012:41289] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/rolllog-proc/abort' 2016-10-20 14:32:55,846 DEBUG [RS:0;cn012:37560] procedure.ZKProcedureMemberRpcs(356): Starting procedure member 'cn012.l42scl.hortonworks.com,37560,1476973974890' 2016-10-20 14:32:55,846 DEBUG [RS:0;cn012:37560] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/rolllog-proc/abort' 2016-10-20 14:32:55,846 DEBUG [M:0;cn012:41289] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/rolllog-proc/acquired' 2016-10-20 14:32:55,846 DEBUG [RS:0;cn012:37560] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/rolllog-proc/acquired' 2016-10-20 14:32:55,847 INFO [M:0;cn012:41289] regionserver.LogRollRegionServerProcedureManager(85): Started region server backup manager. 2016-10-20 14:32:55,847 DEBUG [M:0;cn012:41289] procedure.RegionServerProcedureManagerHost(53): Procedure backup-proc is started 2016-10-20 14:32:55,847 DEBUG [cn012:41289.activeMasterManager] zookeeper.ZKUtil(624): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Unable to get data of znode /1/meta-region-server because node does not exist (not an error) 2016-10-20 14:32:55,847 DEBUG [M:0;cn012:41289] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot is starting 2016-10-20 14:32:55,847 INFO [cn012:41289.activeMasterManager] master.HMaster(942): Re-assigning hbase:meta with replicaId, 0 it was on null 2016-10-20 14:32:55,847 INFO [RS:0;cn012:37560] regionserver.LogRollRegionServerProcedureManager(85): Started region server backup manager. 2016-10-20 14:32:55,848 DEBUG [RS:0;cn012:37560] procedure.RegionServerProcedureManagerHost(53): Procedure backup-proc is started 2016-10-20 14:32:55,848 DEBUG [RS:0;cn012:37560] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot is starting 2016-10-20 14:32:55,848 DEBUG [M:0;cn012:41289] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:55,848 DEBUG [RS:0;cn012:37560] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:32:55,848 DEBUG [M:0;cn012:41289] procedure.ZKProcedureMemberRpcs(356): Starting procedure member 'cn012.l42scl.hortonworks.com,41289,1476973974467' 2016-10-20 14:32:55,849 DEBUG [M:0;cn012:41289] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/online-snapshot/abort' 2016-10-20 14:32:55,849 DEBUG [RS:0;cn012:37560] procedure.ZKProcedureMemberRpcs(356): Starting procedure member 'cn012.l42scl.hortonworks.com,37560,1476973974890' 2016-10-20 14:32:55,849 DEBUG [RS:0;cn012:37560] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/online-snapshot/abort' 2016-10-20 14:32:55,849 DEBUG [M:0;cn012:41289] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/online-snapshot/acquired' 2016-10-20 14:32:55,850 DEBUG [RS:0;cn012:37560] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/online-snapshot/acquired' 2016-10-20 14:32:55,850 DEBUG [M:0;cn012:41289] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot is started 2016-10-20 14:32:55,850 DEBUG [RS:0;cn012:37560] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot is started 2016-10-20 14:32:55,851 DEBUG [cn012:41289.activeMasterManager] master.AssignmentManager(1291): No previous transition plan found (or ignoring an existing plan) for hbase:meta,,1.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=cn012.l42scl.hortonworks.com,41289,1476973974467; 2 (online=2) available servers, forceNewPlan=false 2016-10-20 14:32:55,851 INFO [cn012:41289.activeMasterManager] master.AssignmentManager(1080): Assigning hbase:meta,,1.1588230740 to cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:55,852 INFO [cn012:41289.activeMasterManager] master.RegionStates(1106): Transition {1588230740 state=OFFLINE, ts=1476973975848, server=null} to {1588230740 state=PENDING_OPEN, ts=1476973975851, server=cn012.l42scl.hortonworks.com,41289,1476973974467} 2016-10-20 14:32:55,852 INFO [cn012:41289.activeMasterManager] zookeeper.MetaTableLocator(439): Setting hbase:meta region location in ZooKeeper as cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:55,855 INFO [M:0;cn012:41289] quotas.RegionServerQuotaManager(62): Quota support disabled 2016-10-20 14:32:55,855 INFO [RS:0;cn012:37560] quotas.RegionServerQuotaManager(62): Quota support disabled 2016-10-20 14:32:55,863 DEBUG [cn012:41289.activeMasterManager] zookeeper.MetaTableLocator(451): META region location doesn't exist, create it 2016-10-20 14:32:55,872 DEBUG [cn012:41289.activeMasterManager] master.ServerManager(934): New admin connection to cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:55,891 INFO [cn012:41289.activeMasterManager] regionserver.RSRpcServices(1666): Open hbase:meta,,1.1588230740 2016-10-20 14:32:55,896 INFO [RS_OPEN_META-cn012:41289-0] wal.WALFactory(144): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2016-10-20 14:32:55,915 INFO [RS_OPEN_META-cn012:41289-0] wal.FSHLog(530): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.meta, suffix=.meta, logDir=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467, archiveDir=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs 2016-10-20 14:32:55,943 INFO [RS_OPEN_META-cn012:41289-0] wal.FSHLog(1432): Slow sync cost: 11 ms, current pipeline: [] 2016-10-20 14:32:55,944 INFO [RS_OPEN_META-cn012:41289-0] wal.FSHLog(888): New WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.meta.1476973975915.meta 2016-10-20 14:32:55,952 DEBUG [cn012:41289.activeMasterManager] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476973975902}]}} 2016-10-20 14:32:55,968 DEBUG [RS_OPEN_META-cn012:41289-0] regionserver.HRegion(6339): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2016-10-20 14:32:55,988 DEBUG [RS_OPEN_META-cn012:41289-0] coprocessor.CoprocessorHost(181): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2016-10-20 14:32:55,990 DEBUG [RS_OPEN_META-cn012:41289-0] regionserver.HRegion(7445): Registered coprocessor service: region=hbase:meta,,1 service=hbase.pb.MultiRowMutationService 2016-10-20 14:32:55,994 INFO [RS_OPEN_META-cn012:41289-0] regionserver.RegionCoprocessorHost(376): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2016-10-20 14:32:55,999 DEBUG [RS_OPEN_META-cn012:41289-0] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table meta 1588230740 2016-10-20 14:32:56,000 DEBUG [RS_OPEN_META-cn012:41289-0] regionserver.HRegion(736): Instantiated hbase:meta,,1.1588230740 2016-10-20 14:32:56,007 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:32:56,007 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:32:56,009 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/info 2016-10-20 14:32:56,011 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:32:56,011 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:32:56,013 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/table 2016-10-20 14:32:56,017 DEBUG [RS_OPEN_META-cn012:41289-0] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740 2016-10-20 14:32:56,020 DEBUG [RS_OPEN_META-cn012:41289-0] regionserver.FlushLargeStoresPolicy(72): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table hbase:meta, use config (67108864) instead 2016-10-20 14:32:56,027 DEBUG [RS_OPEN_META-cn012:41289-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/recovered.edits/3.seqid to file, newSeqId=3, maxSeqId=2 2016-10-20 14:32:56,029 INFO [RS_OPEN_META-cn012:41289-0] regionserver.HRegion(871): Onlined 1588230740; next sequenceid=3 2016-10-20 14:32:56,055 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1957): Post open deploy tasks for hbase:meta,,1.1588230740 2016-10-20 14:32:56,066 DEBUG [PostOpenDeployTasks:1588230740] master.AssignmentManager(2884): Got transition OPENED for {1588230740 state=PENDING_OPEN, ts=1476973975851, server=cn012.l42scl.hortonworks.com,41289,1476973974467} from cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:56,067 INFO [PostOpenDeployTasks:1588230740] master.RegionStates(1106): Transition {1588230740 state=PENDING_OPEN, ts=1476973975851, server=cn012.l42scl.hortonworks.com,41289,1476973974467} to {1588230740 state=OPEN, ts=1476973976067, server=cn012.l42scl.hortonworks.com,41289,1476973974467} 2016-10-20 14:32:56,067 INFO [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(439): Setting hbase:meta region location in ZooKeeper as cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:56,079 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/meta-region-server 2016-10-20 14:32:56,080 DEBUG [PostOpenDeployTasks:1588230740] master.RegionStates(452): Onlined 1588230740 on cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:56,083 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1984): Finished post open deploy task for hbase:meta,,1.1588230740 2016-10-20 14:32:56,084 DEBUG [RS_OPEN_META-cn012:41289-0] handler.OpenRegionHandler(126): Opened hbase:meta,,1.1588230740 on cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:56,260 INFO [cn012:41289.activeMasterManager] hbase.MetaTableAccessor(1700): Updated table hbase:meta state to ENABLED in META 2016-10-20 14:32:56,262 DEBUG [cn012:41289.activeMasterManager] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476973976261}]}} 2016-10-20 14:32:56,268 INFO [cn012:41289.activeMasterManager] hbase.MetaTableAccessor(1700): Updated table hbase:meta state to ENABLED in META 2016-10-20 14:32:56,328 DEBUG [cn012:41289.activeMasterManager] procedure.MasterProcedureScheduler(387): Wake event ProcedureEvent(server crash processing) 2016-10-20 14:32:56,329 INFO [cn012:41289.activeMasterManager] master.ServerManager(683): AssignmentManager hasn't finished failover cleanup; waiting 2016-10-20 14:32:56,332 INFO [cn012:41289.activeMasterManager] master.HMaster(969): hbase:meta with replicaId 0 assigned=1, location=cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:56,354 INFO [cn012:41289.activeMasterManager] master.AssignmentManager(555): Clean cluster startup. Don't reassign user regions 2016-10-20 14:32:56,363 INFO [cn012:41289.activeMasterManager] master.AssignmentManager(425): Joined the cluster in 27ms, failover=false 2016-10-20 14:32:56,367 DEBUG [cn012:41289.activeMasterManager] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/info 2016-10-20 14:32:56,370 DEBUG [cn012:41289.activeMasterManager] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/table 2016-10-20 14:32:56,394 INFO [cn012:41289.activeMasterManager] master.TableNamespaceManager(93): Namespace table not found. Creating... 2016-10-20 14:32:56,621 DEBUG [cn012:41289.activeMasterManager] procedure2.ProcedureExecutor(669): Procedure CreateTableProcedure (table=hbase:namespace) id=1 owner=hbase state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2016-10-20 14:32:56,675 DEBUG [ProcedureExecutor-0] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/hbase:namespace/write-master:412890000000000 2016-10-20 14:32:56,873 INFO [IPC Server handler 5 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:32:56,878 INFO [M:0;cn012:41289] wal.FSHLog(530): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0, suffix=, logDir=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467, archiveDir=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs 2016-10-20 14:32:56,880 DEBUG [ProcedureExecutor-0] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2016-10-20 14:32:56,881 INFO [RS:0;cn012:37560] wal.FSHLog(530): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0, suffix=, logDir=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890, archiveDir=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs 2016-10-20 14:32:56,888 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6162): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp Table name == hbase:namespace 2016-10-20 14:32:56,896 INFO [M:0;cn012:41289] wal.FSHLog(1432): Slow sync cost: 12 ms, current pipeline: [] 2016-10-20 14:32:56,899 INFO [M:0;cn012:41289] wal.FSHLog(888): New WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476973976878 2016-10-20 14:32:56,901 INFO [RS:0;cn012:37560] wal.FSHLog(1432): Slow sync cost: 15 ms, current pipeline: [] 2016-10-20 14:32:56,902 INFO [RS:0;cn012:37560] wal.FSHLog(888): New WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476973976881 2016-10-20 14:32:56,918 INFO [IPC Server handler 0 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:32:57,119 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(736): Instantiated hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962. 2016-10-20 14:32:57,122 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1419): Closing hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962.: disabling compactions & flushes 2016-10-20 14:32:57,122 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1446): Updates disabled for region hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962. 2016-10-20 14:32:57,122 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1552): Closed hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962. 2016-10-20 14:32:57,291 DEBUG [ProcedureExecutor-0] hbase.MetaTableAccessor(1374): Put{"totalColumns":1,"row":"hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":9223372036854775807}]}} 2016-10-20 14:32:57,298 INFO [ProcedureExecutor-0] hbase.MetaTableAccessor(1571): Added 1 2016-10-20 14:32:57,452 INFO [ProcedureExecutor-0] master.AssignmentManager(726): Assigning 1 region(s) to cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:57,455 ERROR [ProcedureExecutor-0] master.TableStateManager(134): Unable to get table hbase:namespace state org.apache.hadoop.hbase.TableNotFoundException: hbase:namespace at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546) at org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494) 2016-10-20 14:32:57,461 INFO [ProcedureExecutor-0] master.RegionStates(1106): Transition {b386968767678fdb930a9a40fbb05962 state=OFFLINE, ts=1476973977452, server=null} to {b386968767678fdb930a9a40fbb05962 state=PENDING_OPEN, ts=1476973977461, server=cn012.l42scl.hortonworks.com,41289,1476973974467} 2016-10-20 14:32:57,461 INFO [ProcedureExecutor-0] master.RegionStateStore(207): Updating hbase:meta row hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962. with state=PENDING_OPEN, sn=cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:57,468 INFO [ProcedureExecutor-0] regionserver.RSRpcServices(1666): Open hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962. 2016-10-20 14:32:57,485 DEBUG [ProcedureExecutor-0] master.AssignmentManager(897): Bulk assigning done for cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:57,487 DEBUG [ProcedureExecutor-0] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476973977487}]}} 2016-10-20 14:32:57,494 INFO [ProcedureExecutor-0] hbase.MetaTableAccessor(1700): Updated table hbase:namespace state to ENABLED in META 2016-10-20 14:32:57,496 INFO [RS_OPEN_REGION-cn012:41289-0] wal.FSHLog(530): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1, suffix=, logDir=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467, archiveDir=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs 2016-10-20 14:32:57,514 INFO [RS_OPEN_REGION-cn012:41289-0] wal.FSHLog(1432): Slow sync cost: 12 ms, current pipeline: [] 2016-10-20 14:32:57,516 INFO [RS_OPEN_REGION-cn012:41289-0] wal.FSHLog(888): New WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476973977496 2016-10-20 14:32:57,520 DEBUG [RS_OPEN_REGION-cn012:41289-0] regionserver.HRegion(6339): Opening region: {ENCODED => b386968767678fdb930a9a40fbb05962, NAME => 'hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962.', STARTKEY => '', ENDKEY => ''} 2016-10-20 14:32:57,521 DEBUG [RS_OPEN_REGION-cn012:41289-0] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table namespace b386968767678fdb930a9a40fbb05962 2016-10-20 14:32:57,522 DEBUG [RS_OPEN_REGION-cn012:41289-0] regionserver.HRegion(736): Instantiated hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962. 2016-10-20 14:32:57,529 INFO [StoreOpener-b386968767678fdb930a9a40fbb05962-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:32:57,530 INFO [StoreOpener-b386968767678fdb930a9a40fbb05962-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:32:57,532 DEBUG [StoreOpener-b386968767678fdb930a9a40fbb05962-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/namespace/b386968767678fdb930a9a40fbb05962/info 2016-10-20 14:32:57,535 DEBUG [RS_OPEN_REGION-cn012:41289-0] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/namespace/b386968767678fdb930a9a40fbb05962 2016-10-20 14:32:57,546 DEBUG [RS_OPEN_REGION-cn012:41289-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/namespace/b386968767678fdb930a9a40fbb05962/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:32:57,546 INFO [RS_OPEN_REGION-cn012:41289-0] regionserver.HRegion(871): Onlined b386968767678fdb930a9a40fbb05962; next sequenceid=2 2016-10-20 14:32:57,552 INFO [PostOpenDeployTasks:b386968767678fdb930a9a40fbb05962] regionserver.HRegionServer(1957): Post open deploy tasks for hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962. 2016-10-20 14:32:57,554 DEBUG [PostOpenDeployTasks:b386968767678fdb930a9a40fbb05962] master.AssignmentManager(2884): Got transition OPENED for {b386968767678fdb930a9a40fbb05962 state=PENDING_OPEN, ts=1476973977461, server=cn012.l42scl.hortonworks.com,41289,1476973974467} from cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:57,554 INFO [PostOpenDeployTasks:b386968767678fdb930a9a40fbb05962] master.RegionStates(1106): Transition {b386968767678fdb930a9a40fbb05962 state=PENDING_OPEN, ts=1476973977461, server=cn012.l42scl.hortonworks.com,41289,1476973974467} to {b386968767678fdb930a9a40fbb05962 state=OPEN, ts=1476973977554, server=cn012.l42scl.hortonworks.com,41289,1476973974467} 2016-10-20 14:32:57,555 INFO [PostOpenDeployTasks:b386968767678fdb930a9a40fbb05962] master.RegionStateStore(207): Updating hbase:meta row hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962. with state=OPEN, openSeqNum=2, server=cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:57,560 DEBUG [PostOpenDeployTasks:b386968767678fdb930a9a40fbb05962] master.RegionStates(452): Onlined b386968767678fdb930a9a40fbb05962 on cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:57,562 DEBUG [PostOpenDeployTasks:b386968767678fdb930a9a40fbb05962] regionserver.HRegionServer(1984): Finished post open deploy task for hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962. 2016-10-20 14:32:57,562 DEBUG [RS_OPEN_REGION-cn012:41289-0] handler.OpenRegionHandler(126): Opened hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962. on cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:32:57,621 DEBUG [cn012:41289.activeMasterManager] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/namespace 2016-10-20 14:32:57,638 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/namespace 2016-10-20 14:32:57,802 DEBUG [cn012:41289.activeMasterManager] procedure2.ProcedureExecutor(669): Procedure CreateNamespaceProcedure (Namespace=default) id=2 owner=hbase state=RUNNABLE:CREATE_NAMESPACE_PREPARE added to the store. 2016-10-20 14:32:57,963 DEBUG [ProcedureExecutor-0] lock.ZKInterProcessLockBase(328): Released /1/table-lock/hbase:namespace/write-master:412890000000000 2016-10-20 14:32:57,963 DEBUG [ProcedureExecutor-0] procedure2.ProcedureExecutor(870): Procedure completed in 1.4020sec: CreateTableProcedure (table=hbase:namespace) id=1 owner=hbase state=FINISHED 2016-10-20 14:32:58,446 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/namespace 2016-10-20 14:32:58,463 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node default with data: \x0A\x07default 2016-10-20 14:32:58,984 DEBUG [ProcedureExecutor-0] procedure2.ProcedureExecutor(870): Procedure completed in 1.1790sec: CreateNamespaceProcedure (Namespace=default) id=2 owner=hbase state=FINISHED 2016-10-20 14:32:59,169 DEBUG [cn012:41289.activeMasterManager] procedure2.ProcedureExecutor(669): Procedure CreateNamespaceProcedure (Namespace=hbase) id=3 owner=hbase state=RUNNABLE:CREATE_NAMESPACE_PREPARE added to the store. 2016-10-20 14:32:59,671 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/namespace 2016-10-20 14:32:59,682 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node default with data: \x0A\x07default 2016-10-20 14:32:59,683 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node hbase with data: \x0A\x05hbase 2016-10-20 14:32:59,992 DEBUG [ProcedureExecutor-1] procedure2.ProcedureExecutor(870): Procedure completed in 844msec: CreateNamespaceProcedure (Namespace=hbase) id=3 owner=hbase state=FINISHED 2016-10-20 14:33:00,021 DEBUG [cn012:41289.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /1/namespace/default already exists 2016-10-20 14:33:00,030 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/namespace/default 2016-10-20 14:33:00,038 DEBUG [cn012:41289.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /1/namespace/hbase already exists 2016-10-20 14:33:00,046 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/1/namespace/hbase 2016-10-20 14:33:00,047 INFO [cn012:41289.activeMasterManager] master.HMaster(811): Master has completed initialization 2016-10-20 14:33:00,047 DEBUG [cn012:41289.activeMasterManager] procedure.MasterProcedureScheduler(387): Wake event ProcedureEvent(master initialized) 2016-10-20 14:33:00,052 INFO [cn012:41289.activeMasterManager] quotas.MasterQuotaManager(72): Quota support disabled 2016-10-20 14:33:00,052 INFO [cn012:41289.activeMasterManager] zookeeper.ZooKeeperWatcher(225): not a secure deployment, proceeding 2016-10-20 14:33:00,100 INFO [cn012:41289.activeMasterManager] master.HMaster(1501): Client=null/null create 'hbase:backup', {NAME => 'meta', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'session', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2016-10-20 14:33:00,122 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x7f829c76 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:33:00,130 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x7f829c760x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:33:00,133 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x7f829c76-0x157e2823ec60006 connected 2016-10-20 14:33:00,134 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c3ebc6b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:33:00,134 DEBUG [main] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:33:00,134 DEBUG [main] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:33:00,143 DEBUG [AsyncRpcChannel-pool2-t2] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:33:00,144 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42401; # active connections: 2 2016-10-20 14:33:00,147 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:33:00,149 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42401 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:00,182 INFO [main] hbase.HBaseTestingUtility(1090): Minicluster is up 2016-10-20 14:33:00,183 INFO [main] hbase.HBaseTestingUtility(1264): The hbase.fs.tmp.dir is set to /user/hbase/hbase-staging 2016-10-20 14:33:00,183 INFO [main] hbase.HBaseTestingUtility(1014): Starting up minicluster with 1 master(s) and 1 regionserver(s) and 1 datanode(s) 2016-10-20 14:33:00,217 INFO [main] hbase.HBaseTestingUtility(429): System.getProperty("hadoop.log.dir") already set to: /mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/hadoop_logs so I do NOT create it in target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99 2016-10-20 14:33:00,217 WARN [main] hbase.HBaseTestingUtility(433): hadoop.log.dir property value differs in configuration and system: Configuration=/mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/hadoop-log-dir while System=/mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/hadoop_logs Erasing configuration value by system value. 2016-10-20 14:33:00,217 INFO [main] hbase.HBaseTestingUtility(429): System.getProperty("hadoop.tmp.dir") already set to: /mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/hadoop_tmp so I do NOT create it in target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99 2016-10-20 14:33:00,217 WARN [main] hbase.HBaseTestingUtility(433): hadoop.tmp.dir property value differs in configuration and system: Configuration=/mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/hadoop-tmp-dir while System=/mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/hadoop_tmp Erasing configuration value by system value. 2016-10-20 14:33:00,217 INFO [main] hbase.HBaseTestingUtility(497): Created new mini-cluster data directory: /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/dfscluster_0567527d-a565-4479-bfb0-c584662a6b9b, deleteOnExit=true 2016-10-20 14:33:00,218 INFO [main] hbase.HBaseTestingUtility(744): Setting test.cache.data to /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/cache_data in system properties and HBase conf 2016-10-20 14:33:00,218 INFO [main] hbase.HBaseTestingUtility(744): Setting hadoop.tmp.dir to /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop_tmp in system properties and HBase conf 2016-10-20 14:33:00,218 INFO [main] hbase.HBaseTestingUtility(744): Setting hadoop.log.dir to /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop_logs in system properties and HBase conf 2016-10-20 14:33:00,219 INFO [main] hbase.HBaseTestingUtility(744): Setting mapreduce.cluster.local.dir to /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/mapred_local in system properties and HBase conf 2016-10-20 14:33:00,219 INFO [main] hbase.HBaseTestingUtility(744): Setting mapreduce.cluster.temp.dir to /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/mapred_temp in system properties and HBase conf 2016-10-20 14:33:00,219 INFO [main] hbase.HBaseTestingUtility(735): read short circuit is OFF 2016-10-20 14:33:00,220 DEBUG [main] fs.HFileSystem(221): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2016-10-20 14:33:00,251 DEBUG [cn012:41289.activeMasterManager] procedure2.ProcedureExecutor(669): Procedure CreateTableProcedure (table=hbase:backup) id=4 owner=hbase state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2016-10-20 14:33:00,410 DEBUG [ProcedureExecutor-3] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/hbase:backup/write-master:412890000000000 2016-10-20 14:33:00,413 INFO [cn012:41289.activeMasterManager] master.BackupController(51): Created hbase:backup table 2016-10-20 14:33:00,540 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2016-10-20 14:33:00,547 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/hbase/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.1/hadoop-hdfs-2.7.1-tests.jar!/webapps/hdfs to /tmp/Jetty_localhost_43243_hdfs____uwovss/webapp 2016-10-20 14:33:00,600 INFO [IPC Server handler 0 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741836_1012{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 535 2016-10-20 14:33:00,697 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43243 2016-10-20 14:33:00,978 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2016-10-20 14:33:00,983 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/hbase/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.1/hadoop-hdfs-2.7.1-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_38468_datanode____4zil7c/webapp 2016-10-20 14:33:01,018 DEBUG [ProcedureExecutor-3] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/hbase/backup/.tabledesc/.tableinfo.0000000001 2016-10-20 14:33:01,020 INFO [RegionOpenAndInitThread-hbase:backup-1] regionserver.HRegion(6162): creating HRegion hbase:backup HTD == 'hbase:backup', {NAME => 'meta', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'session', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp Table name == hbase:backup 2016-10-20 14:33:01,037 INFO [IPC Server handler 1 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:01,038 DEBUG [RegionOpenAndInitThread-hbase:backup-1] regionserver.HRegion(736): Instantiated hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0. 2016-10-20 14:33:01,039 DEBUG [RegionOpenAndInitThread-hbase:backup-1] regionserver.HRegion(1419): Closing hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0.: disabling compactions & flushes 2016-10-20 14:33:01,039 DEBUG [RegionOpenAndInitThread-hbase:backup-1] regionserver.HRegion(1446): Updates disabled for region hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0. 2016-10-20 14:33:01,039 INFO [RegionOpenAndInitThread-hbase:backup-1] regionserver.HRegion(1552): Closed hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0. 2016-10-20 14:33:01,123 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38468 2016-10-20 14:33:01,237 DEBUG [ProcedureExecutor-3] hbase.MetaTableAccessor(1374): Put{"totalColumns":1,"row":"hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0.","families":{"info":[{"qualifier":"regioninfo","vlen":38,"tag":[],"timestamp":9223372036854775807}]}} 2016-10-20 14:33:01,241 INFO [ProcedureExecutor-3] hbase.MetaTableAccessor(1571): Added 1 2016-10-20 14:33:01,427 INFO [ProcedureExecutor-3] master.AssignmentManager(726): Assigning 1 region(s) to cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:01,428 ERROR [ProcedureExecutor-3] master.TableStateManager(134): Unable to get table hbase:backup state org.apache.hadoop.hbase.TableNotFoundException: hbase:backup at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546) at org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494) 2016-10-20 14:33:01,429 INFO [ProcedureExecutor-3] master.RegionStates(1106): Transition {7b31f6434ce41b4dbce476e8d5a763e0 state=OFFLINE, ts=1476973981426, server=null} to {7b31f6434ce41b4dbce476e8d5a763e0 state=PENDING_OPEN, ts=1476973981429, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:33:01,429 INFO [ProcedureExecutor-3] master.RegionStateStore(207): Updating hbase:meta row hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0. with state=PENDING_OPEN, sn=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:01,432 DEBUG [ProcedureExecutor-3] master.ServerManager(934): New admin connection to cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:01,439 DEBUG [AsyncRpcChannel-pool2-t3] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service AdminService, sasl=false 2016-10-20 14:33:01,442 DEBUG [RpcServer.listener,port=37560] ipc.RpcServer$Listener(880): RpcServer.listener,port=37560: connection from 172.18.128.12:41784; # active connections: 1 2016-10-20 14:33:01,443 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:33:01,444 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 41784 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:01,446 INFO [PriorityRpcServer.handler=0,queue=0,port=37560] regionserver.RSRpcServices(1666): Open hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0. 2016-10-20 14:33:01,460 DEBUG [ProcedureExecutor-3] master.AssignmentManager(897): Bulk assigning done for cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:01,461 DEBUG [ProcedureExecutor-3] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"hbase:backup","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476973981460}]}} 2016-10-20 14:33:01,465 INFO [ProcedureExecutor-3] hbase.MetaTableAccessor(1700): Updated table hbase:backup state to ENABLED in META 2016-10-20 14:33:01,466 INFO [RS_OPEN_REGION-cn012:37560-0] wal.FSHLog(530): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1, suffix=, logDir=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890, archiveDir=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs 2016-10-20 14:33:01,484 INFO [RS_OPEN_REGION-cn012:37560-0] wal.FSHLog(1432): Slow sync cost: 11 ms, current pipeline: [] 2016-10-20 14:33:01,486 INFO [RS_OPEN_REGION-cn012:37560-0] wal.FSHLog(888): New WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476973981466 2016-10-20 14:33:01,489 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(6339): Opening region: {ENCODED => 7b31f6434ce41b4dbce476e8d5a763e0, NAME => 'hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0.', STARTKEY => '', ENDKEY => ''} 2016-10-20 14:33:01,490 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table backup 7b31f6434ce41b4dbce476e8d5a763e0 2016-10-20 14:33:01,491 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(736): Instantiated hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0. 2016-10-20 14:33:01,500 INFO [StoreOpener-7b31f6434ce41b4dbce476e8d5a763e0-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:01,501 INFO [StoreOpener-7b31f6434ce41b4dbce476e8d5a763e0-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:33:01,503 DEBUG [StoreOpener-7b31f6434ce41b4dbce476e8d5a763e0-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/backup/7b31f6434ce41b4dbce476e8d5a763e0/meta 2016-10-20 14:33:01,507 INFO [StoreOpener-7b31f6434ce41b4dbce476e8d5a763e0-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:01,508 INFO [StoreOpener-7b31f6434ce41b4dbce476e8d5a763e0-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:33:01,510 DEBUG [StoreOpener-7b31f6434ce41b4dbce476e8d5a763e0-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/backup/7b31f6434ce41b4dbce476e8d5a763e0/session 2016-10-20 14:33:01,511 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/backup/7b31f6434ce41b4dbce476e8d5a763e0 2016-10-20 14:33:01,516 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.FlushLargeStoresPolicy(72): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table hbase:backup, use config (67108864) instead 2016-10-20 14:33:01,517 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2016-10-20 14:33:01,530 DEBUG [RS_OPEN_REGION-cn012:37560-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/backup/7b31f6434ce41b4dbce476e8d5a763e0/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:33:01,530 INFO [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(871): Onlined 7b31f6434ce41b4dbce476e8d5a763e0; next sequenceid=2 2016-10-20 14:33:01,534 INFO [PostOpenDeployTasks:7b31f6434ce41b4dbce476e8d5a763e0] regionserver.HRegionServer(1957): Post open deploy tasks for hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0. 2016-10-20 14:33:01,538 DEBUG [PriorityRpcServer.handler=1,queue=1,port=41289] master.AssignmentManager(2884): Got transition OPENED for {7b31f6434ce41b4dbce476e8d5a763e0 state=PENDING_OPEN, ts=1476973981429, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:01,538 INFO [PriorityRpcServer.handler=1,queue=1,port=41289] master.RegionStates(1106): Transition {7b31f6434ce41b4dbce476e8d5a763e0 state=PENDING_OPEN, ts=1476973981429, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {7b31f6434ce41b4dbce476e8d5a763e0 state=OPEN, ts=1476973981538, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:33:01,539 INFO [PriorityRpcServer.handler=1,queue=1,port=41289] master.RegionStateStore(207): Updating hbase:meta row hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0. with state=OPEN, openSeqNum=2, server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:01,541 DEBUG [PriorityRpcServer.handler=1,queue=1,port=41289] master.RegionStates(452): Onlined 7b31f6434ce41b4dbce476e8d5a763e0 on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:01,543 DEBUG [PostOpenDeployTasks:7b31f6434ce41b4dbce476e8d5a763e0] regionserver.HRegionServer(1984): Finished post open deploy task for hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0. 2016-10-20 14:33:01,546 DEBUG [RS_OPEN_REGION-cn012:37560-0] handler.OpenRegionHandler(126): Opened hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0. on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:01,767 INFO [IPC Server handler 9 on 46268] blockmanagement.BlockManager(1862): BLOCK* processReport: from storage DS-651c82c6-88ce-4f3b-bae6-e3460c66df10 node DatanodeRegistration(127.0.0.1:43453, datanodeUuid=fe6ec94c-4b95-4177-bb8b-a2034fdb7a43, infoPort=44303, infoSecurePort=0, ipcPort=42289, storageInfo=lv=-56;cid=testClusterID;nsid=1126480624;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs 2016-10-20 14:33:01,768 INFO [IPC Server handler 9 on 46268] blockmanagement.BlockManager(1862): BLOCK* processReport: from storage DS-a82af92a-d58e-4076-affe-976f93711253 node DatanodeRegistration(127.0.0.1:43453, datanodeUuid=fe6ec94c-4b95-4177-bb8b-a2034fdb7a43, infoPort=44303, infoSecurePort=0, ipcPort=42289, storageInfo=lv=-56;cid=testClusterID;nsid=1126480624;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2016-10-20 14:33:01,778 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-20 14:33:01,781 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-20 14:33:01,820 INFO [IPC Server handler 8 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-651c82c6-88ce-4f3b-bae6-e3460c66df10:NORMAL:127.0.0.1:43453|RBW]]} size 0 2016-10-20 14:33:01,826 INFO [main] util.FSUtils(749): Created version file at hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998 with version=8 2016-10-20 14:33:01,829 DEBUG [main] impl.BackupManager(160): Added region procedure manager: org.apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureManager 2016-10-20 14:33:01,829 INFO [main] client.ConnectionUtils(106): master/cn012.l42scl.hortonworks.com/172.18.128.12:0 server-side HConnection retries=350 2016-10-20 14:33:01,830 INFO [main] ipc.SimpleRpcScheduler(190): Using deadline as user call queue, count=1 2016-10-20 14:33:01,831 INFO [main] ipc.RpcServer$Listener(635): master/cn012.l42scl.hortonworks.com/172.18.128.12:0: started 3 reader(s) listening on port=34094 2016-10-20 14:33:01,831 INFO [main] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:01,832 INFO [main] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:01,833 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-20 14:33:01,834 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=master:34094 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:33:01,871 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:340940x0, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:33:01,874 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): master:34094-0x157e2823ec60007 connected 2016-10-20 14:33:01,938 DEBUG [main] zookeeper.ZKUtil(367): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on znode that does not yet exist, /2/master 2016-10-20 14:33:01,939 DEBUG [main] zookeeper.ZKUtil(367): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on znode that does not yet exist, /2/running 2016-10-20 14:33:01,940 INFO [RpcServer.responder] ipc.RpcServer$Responder(958): RpcServer.responder: starting 2016-10-20 14:33:01,940 INFO [RpcServer.listener,port=34094] ipc.RpcServer$Listener(769): RpcServer.listener,port=34094: starting 2016-10-20 14:33:01,940 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=0 queue=0 2016-10-20 14:33:01,942 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=1 queue=0 2016-10-20 14:33:01,942 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=2 queue=0 2016-10-20 14:33:01,942 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=3 queue=0 2016-10-20 14:33:01,943 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=4 queue=0 2016-10-20 14:33:01,943 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=0 queue=0 2016-10-20 14:33:01,943 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=1 queue=1 2016-10-20 14:33:01,944 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=2 queue=0 2016-10-20 14:33:01,944 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=3 queue=1 2016-10-20 14:33:01,944 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=4 queue=0 2016-10-20 14:33:01,945 DEBUG [main] ipc.RpcExecutor(118): Replication Start Handler index=0 queue=0 2016-10-20 14:33:01,945 DEBUG [main] ipc.RpcExecutor(118): Replication Start Handler index=1 queue=0 2016-10-20 14:33:01,945 DEBUG [main] ipc.RpcExecutor(118): Replication Start Handler index=2 queue=0 2016-10-20 14:33:01,946 INFO [main] master.HMaster(401): hbase.rootdir=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998, hbase.cluster.distributed=false 2016-10-20 14:33:01,947 DEBUG [main] impl.BackupManager(136): Added log cleaner: org.apache.hadoop.hbase.backup.master.BackupLogCleaner 2016-10-20 14:33:01,947 DEBUG [main] impl.BackupManager(137): Added master procedure manager: org.apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager 2016-10-20 14:33:01,947 DEBUG [main] impl.BackupManager(138): Added master observer: org.apache.hadoop.hbase.backup.master.BackupController 2016-10-20 14:33:01,948 INFO [main] master.HMaster(1725): Adding backup master ZNode /2/backup-masters/cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:01,956 DEBUG [main] zookeeper.ZKUtil(365): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on existing znode=/2/backup-masters/cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:01,963 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/2/master 2016-10-20 14:33:01,965 DEBUG [cn012:34094.activeMasterManager] zookeeper.ZKUtil(365): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on existing znode=/2/master 2016-10-20 14:33:01,967 DEBUG [main-EventThread] zookeeper.ZKUtil(365): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on existing znode=/2/master 2016-10-20 14:33:01,967 INFO [cn012:34094.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /2/backup-masters/cn012.l42scl.hortonworks.com,34094,1476973981831 from backup master directory 2016-10-20 14:33:01,967 DEBUG [main-EventThread] master.ActiveMasterManager(126): A master is now available 2016-10-20 14:33:01,971 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/backup-masters/cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:01,971 WARN [cn012:34094.activeMasterManager] hbase.ZNodeClearer(61): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2016-10-20 14:33:01,971 INFO [cn012:34094.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:01,997 DEBUG [main] impl.BackupManager(160): Added region procedure manager: org.apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureManager 2016-10-20 14:33:01,997 INFO [main] client.ConnectionUtils(106): regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0 server-side HConnection retries=350 2016-10-20 14:33:01,998 INFO [main] ipc.SimpleRpcScheduler(190): Using deadline as user call queue, count=1 2016-10-20 14:33:02,004 INFO [main] ipc.RpcServer$Listener(635): regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0: started 3 reader(s) listening on port=34206 2016-10-20 14:33:02,007 INFO [main] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:02,008 INFO [main] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:02,013 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-20 14:33:02,014 INFO [IPC Server handler 2 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741826_1002{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a82af92a-d58e-4076-affe-976f93711253:NORMAL:127.0.0.1:43453|RBW]]} size 42 2016-10-20 14:33:02,017 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:34206 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:33:02,038 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:342060x0, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:33:02,039 DEBUG [main] zookeeper.ZKUtil(365): regionserver:342060x0, quorum=localhost:56750, baseZNode=/2 Set watcher on existing znode=/2/master 2016-10-20 14:33:02,040 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): regionserver:34206-0x157e2823ec60008 connected 2016-10-20 14:33:02,046 DEBUG [ProcedureExecutor-3] lock.ZKInterProcessLockBase(328): Released /1/table-lock/hbase:backup/write-master:412890000000000 2016-10-20 14:33:02,046 DEBUG [ProcedureExecutor-3] procedure2.ProcedureExecutor(870): Procedure completed in 1.7700sec: CreateTableProcedure (table=hbase:backup) id=4 owner=hbase state=FINISHED 2016-10-20 14:33:02,047 DEBUG [main] zookeeper.ZKUtil(367): regionserver:34206-0x157e2823ec60008, quorum=localhost:56750, baseZNode=/2 Set watcher on znode that does not yet exist, /2/running 2016-10-20 14:33:02,047 INFO [RpcServer.responder] ipc.RpcServer$Responder(958): RpcServer.responder: starting 2016-10-20 14:33:02,047 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=0 queue=0 2016-10-20 14:33:02,047 INFO [RpcServer.listener,port=34206] ipc.RpcServer$Listener(769): RpcServer.listener,port=34206: starting 2016-10-20 14:33:02,048 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=1 queue=0 2016-10-20 14:33:02,048 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=2 queue=0 2016-10-20 14:33:02,049 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=3 queue=0 2016-10-20 14:33:02,049 DEBUG [main] ipc.RpcExecutor(118): B.default Start Handler index=4 queue=0 2016-10-20 14:33:02,049 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=0 queue=0 2016-10-20 14:33:02,049 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=1 queue=1 2016-10-20 14:33:02,050 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=2 queue=0 2016-10-20 14:33:02,050 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=3 queue=1 2016-10-20 14:33:02,050 DEBUG [main] ipc.RpcExecutor(118): Priority Start Handler index=4 queue=0 2016-10-20 14:33:02,050 DEBUG [main] ipc.RpcExecutor(118): Replication Start Handler index=0 queue=0 2016-10-20 14:33:02,051 DEBUG [main] ipc.RpcExecutor(118): Replication Start Handler index=1 queue=0 2016-10-20 14:33:02,051 DEBUG [main] ipc.RpcExecutor(118): Replication Start Handler index=2 queue=0 2016-10-20 14:33:02,054 INFO [M:0;cn012:34094] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x51cb701a connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:33:02,054 INFO [RS:0;cn012:34206] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x45d062f2 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:33:02,063 DEBUG [M:0;cn012:34094-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x51cb701a0x0, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:33:02,064 DEBUG [M:0;cn012:34094-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x51cb701a-0x157e2823ec60009 connected 2016-10-20 14:33:02,071 DEBUG [RS:0;cn012:34206-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x45d062f20x0, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:33:02,071 INFO [M:0;cn012:34094] client.ZooKeeperRegistry(104): ClusterId read in ZooKeeper is null 2016-10-20 14:33:02,071 DEBUG [M:0;cn012:34094] client.ConnectionImplementation(466): clusterid came back null, using default default-cluster 2016-10-20 14:33:02,071 INFO [RS:0;cn012:34206] client.ZooKeeperRegistry(104): ClusterId read in ZooKeeper is null 2016-10-20 14:33:02,072 DEBUG [M:0;cn012:34094] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f0c9453, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:33:02,072 DEBUG [M:0;cn012:34094] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:33:02,072 DEBUG [RS:0;cn012:34206] client.ConnectionImplementation(466): clusterid came back null, using default default-cluster 2016-10-20 14:33:02,072 DEBUG [M:0;cn012:34094] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:33:02,072 DEBUG [RS:0;cn012:34206] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b178438, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:33:02,072 DEBUG [RS:0;cn012:34206] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:33:02,073 DEBUG [RS:0;cn012:34206] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:33:02,073 DEBUG [RS:0;cn012:34206-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x45d062f2-0x157e2823ec6000a connected 2016-10-20 14:33:02,425 DEBUG [cn012:34094.activeMasterManager] util.FSUtils(901): Created cluster ID file at hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/hbase.id with ID: a5c2b2bf-9dd5-42bd-b1ca-1a735489928b 2016-10-20 14:33:02,439 INFO [cn012:34094.activeMasterManager] master.MasterFileSystem(528): BOOTSTRAP: creating hbase:meta region 2016-10-20 14:33:02,440 INFO [cn012:34094.activeMasterManager] regionserver.HRegion(6162): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', BLOOMFILTER => 'NONE', VERSIONS => '3', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'false', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998 Table name == hbase:meta 2016-10-20 14:33:02,465 INFO [IPC Server handler 3 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741827_1003{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-651c82c6-88ce-4f3b-bae6-e3460c66df10:NORMAL:127.0.0.1:43453|RBW]]} size 32 2016-10-20 14:33:02,868 DEBUG [cn012:34094.activeMasterManager] regionserver.HRegion(736): Instantiated hbase:meta,,1.1588230740 2016-10-20 14:33:02,878 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:02,879 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:33:02,885 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/info 2016-10-20 14:33:02,889 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:02,890 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:33:02,892 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/table 2016-10-20 14:33:02,896 DEBUG [cn012:34094.activeMasterManager] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740 2016-10-20 14:33:02,902 DEBUG [cn012:34094.activeMasterManager] regionserver.FlushLargeStoresPolicy(72): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table hbase:meta, use config (67108864) instead 2016-10-20 14:33:02,912 DEBUG [cn012:34094.activeMasterManager] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:33:02,912 INFO [cn012:34094.activeMasterManager] regionserver.HRegion(871): Onlined 1588230740; next sequenceid=2 2016-10-20 14:33:02,912 DEBUG [cn012:34094.activeMasterManager] regionserver.HRegion(1419): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2016-10-20 14:33:02,912 DEBUG [cn012:34094.activeMasterManager] regionserver.HRegion(1446): Updates disabled for region hbase:meta,,1.1588230740 2016-10-20 14:33:02,913 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(839): Closed info 2016-10-20 14:33:02,913 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(839): Closed table 2016-10-20 14:33:02,913 INFO [cn012:34094.activeMasterManager] regionserver.HRegion(1552): Closed hbase:meta,,1.1588230740 2016-10-20 14:33:02,938 INFO [IPC Server handler 5 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741828_1004{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a82af92a-d58e-4076-affe-976f93711253:NORMAL:127.0.0.1:43453|RBW]]} size 672 2016-10-20 14:33:03,345 DEBUG [cn012:34094.activeMasterManager] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2016-10-20 14:33:03,366 INFO [cn012:34094.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-10-20 14:33:03,370 INFO [cn012:34094.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2016-10-20 14:33:03,371 DEBUG [cn012:34094.activeMasterManager] util.FSTableDescriptors(222): Fetching table descriptors from the filesystem. 2016-10-20 14:33:03,390 INFO [cn012:34094.activeMasterManager] balancer.StochasticLoadBalancer(156): loading config 2016-10-20 14:33:03,392 DEBUG [cn012:34094.activeMasterManager] zookeeper.ZKUtil(367): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on znode that does not yet exist, /2/balancer 2016-10-20 14:33:03,393 DEBUG [cn012:34094.activeMasterManager] zookeeper.ZKUtil(367): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on znode that does not yet exist, /2/normalizer 2016-10-20 14:33:03,406 DEBUG [cn012:34094.activeMasterManager] zookeeper.ZKUtil(367): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on znode that does not yet exist, /2/switch/split 2016-10-20 14:33:03,407 DEBUG [cn012:34094.activeMasterManager] zookeeper.ZKUtil(367): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on znode that does not yet exist, /2/switch/merge 2016-10-20 14:33:03,413 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/2/running 2016-10-20 14:33:03,413 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:34206-0x157e2823ec60008, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/2/running 2016-10-20 14:33:03,414 INFO [cn012:34094.activeMasterManager] master.HMaster(624): Server active/primary master=cn012.l42scl.hortonworks.com,34094,1476973981831, sessionid=0x157e2823ec60007, setting cluster-up flag (Was=false) 2016-10-20 14:33:03,417 INFO [cn012:34094.activeMasterManager] procedure.ProcedureManagerHost(71): User procedure org.apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager was loaded successfully. 2016-10-20 14:33:03,421 INFO [M:0;cn012:34094] regionserver.HRegionServer(813): ClusterId : a5c2b2bf-9dd5-42bd-b1ca-1a735489928b 2016-10-20 14:33:03,421 INFO [RS:0;cn012:34206] regionserver.HRegionServer(813): ClusterId : a5c2b2bf-9dd5-42bd-b1ca-1a735489928b 2016-10-20 14:33:03,421 INFO [M:0;cn012:34094] procedure.ProcedureManagerHost(71): User procedure org.apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureManager was loaded successfully. 2016-10-20 14:33:03,421 INFO [RS:0;cn012:34206] procedure.ProcedureManagerHost(71): User procedure org.apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureManager was loaded successfully. 2016-10-20 14:33:03,422 DEBUG [M:0;cn012:34094] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc is initializing 2016-10-20 14:33:03,422 DEBUG [RS:0;cn012:34206] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc is initializing 2016-10-20 14:33:03,438 DEBUG [RS:0;cn012:34206] zookeeper.RecoverableZooKeeper(594): Node /2/flush-table-proc/acquired already exists 2016-10-20 14:33:03,446 DEBUG [cn012:34094.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /2/flush-table-proc/acquired already exists 2016-10-20 14:33:03,455 DEBUG [RS:0;cn012:34206] zookeeper.RecoverableZooKeeper(594): Node /2/flush-table-proc/reached already exists 2016-10-20 14:33:03,463 DEBUG [M:0;cn012:34094] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc is initialized 2016-10-20 14:33:03,463 DEBUG [M:0;cn012:34094] procedure.RegionServerProcedureManagerHost(43): Procedure backup-proc is initializing 2016-10-20 14:33:03,471 DEBUG [cn012:34094.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /2/flush-table-proc/abort already exists 2016-10-20 14:33:03,471 DEBUG [RS:0;cn012:34206] zookeeper.RecoverableZooKeeper(594): Node /2/flush-table-proc/abort already exists 2016-10-20 14:33:03,471 INFO [cn012:34094.activeMasterManager] procedure.ZKProcedureUtil(270): Clearing all procedure znodes: /2/flush-table-proc/acquired /2/flush-table-proc/reached /2/flush-table-proc/abort 2016-10-20 14:33:03,472 DEBUG [RS:0;cn012:34206] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc is initialized 2016-10-20 14:33:03,472 DEBUG [RS:0;cn012:34206] procedure.RegionServerProcedureManagerHost(43): Procedure backup-proc is initializing 2016-10-20 14:33:03,504 DEBUG [RS:0;cn012:34206] zookeeper.RecoverableZooKeeper(594): Node /2/rolllog-proc already exists 2016-10-20 14:33:03,514 DEBUG [cn012:34094.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(248): Starting the controller for procedure member:cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,521 DEBUG [RS:0;cn012:34206] zookeeper.RecoverableZooKeeper(594): Node /2/rolllog-proc/acquired already exists 2016-10-20 14:33:03,529 DEBUG [cn012:34094.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /2/rolllog-proc/acquired already exists 2016-10-20 14:33:03,546 DEBUG [M:0;cn012:34094] procedure.RegionServerProcedureManagerHost(45): Procedure backup-proc is initialized 2016-10-20 14:33:03,546 DEBUG [cn012:34094.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /2/rolllog-proc/abort already exists 2016-10-20 14:33:03,546 DEBUG [M:0;cn012:34094] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot is initializing 2016-10-20 14:33:03,547 INFO [cn012:34094.activeMasterManager] procedure.ZKProcedureUtil(270): Clearing all procedure znodes: /2/rolllog-proc/acquired /2/rolllog-proc/reached /2/rolllog-proc/abort 2016-10-20 14:33:03,554 DEBUG [RS:0;cn012:34206] zookeeper.RecoverableZooKeeper(594): Node /2/rolllog-proc/abort already exists 2016-10-20 14:33:03,555 DEBUG [RS:0;cn012:34206] procedure.RegionServerProcedureManagerHost(45): Procedure backup-proc is initialized 2016-10-20 14:33:03,555 DEBUG [RS:0;cn012:34206] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot is initializing 2016-10-20 14:33:03,588 DEBUG [RS:0;cn012:34206] zookeeper.RecoverableZooKeeper(594): Node /2/online-snapshot already exists 2016-10-20 14:33:03,597 DEBUG [cn012:34094.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(248): Starting the controller for procedure member:cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,604 DEBUG [RS:0;cn012:34206] zookeeper.RecoverableZooKeeper(594): Node /2/online-snapshot/acquired already exists 2016-10-20 14:33:03,613 DEBUG [cn012:34094.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /2/online-snapshot/acquired already exists 2016-10-20 14:33:03,654 DEBUG [RS:0;cn012:34206] zookeeper.RecoverableZooKeeper(594): Node /2/online-snapshot/abort already exists 2016-10-20 14:33:03,654 DEBUG [M:0;cn012:34094] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot is initialized 2016-10-20 14:33:03,655 DEBUG [RS:0;cn012:34206] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot is initialized 2016-10-20 14:33:03,655 INFO [M:0;cn012:34094] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2016-10-20 14:33:03,655 INFO [RS:0;cn012:34206] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2016-10-20 14:33:03,656 INFO [M:0;cn012:34094] throttle.PressureAwareCompactionThroughputController(132): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2016-10-20 14:33:03,656 INFO [RS:0;cn012:34206] throttle.PressureAwareCompactionThroughputController(132): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2016-10-20 14:33:03,656 INFO [M:0;cn012:34094] regionserver.HRegionServer$CompactionChecker(1555): CompactionChecker runs every 1sec 2016-10-20 14:33:03,657 INFO [RS:0;cn012:34206] regionserver.HRegionServer$CompactionChecker(1555): CompactionChecker runs every 1sec 2016-10-20 14:33:03,657 DEBUG [M:0;cn012:34094] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14f07614, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=cn012.l42scl.hortonworks.com/172.18.128.12:0 2016-10-20 14:33:03,657 DEBUG [RS:0;cn012:34206] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cabba46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=cn012.l42scl.hortonworks.com/172.18.128.12:0 2016-10-20 14:33:03,657 DEBUG [M:0;cn012:34094] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:33:03,657 DEBUG [RS:0;cn012:34206] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:33:03,657 DEBUG [M:0;cn012:34094] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:33:03,657 DEBUG [RS:0;cn012:34206] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:33:03,658 DEBUG [M:0;cn012:34094] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:M:0;cn012:34094 2016-10-20 14:33:03,658 DEBUG [RS:0;cn012:34206] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:RS:0;cn012:34206 2016-10-20 14:33:03,663 DEBUG [cn012:34094.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /2/online-snapshot/abort already exists 2016-10-20 14:33:03,663 INFO [cn012:34094.activeMasterManager] procedure.ZKProcedureUtil(270): Clearing all procedure znodes: /2/online-snapshot/acquired /2/online-snapshot/reached /2/online-snapshot/abort 2016-10-20 14:33:03,671 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/2/rs 2016-10-20 14:33:03,673 DEBUG [RS:0;cn012:34206] zookeeper.ZKUtil(365): regionserver:34206-0x157e2823ec60008, quorum=localhost:56750, baseZNode=/2 Set watcher on existing znode=/2/rs/cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:03,673 DEBUG [M:0;cn012:34094] zookeeper.ZKUtil(365): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on existing znode=/2/rs/cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,674 INFO [RS:0;cn012:34206] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2016-10-20 14:33:03,674 INFO [RS:0;cn012:34206] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2016-10-20 14:33:03,674 INFO [M:0;cn012:34094] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2016-10-20 14:33:03,674 INFO [M:0;cn012:34094] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2016-10-20 14:33:03,674 INFO [RS:0;cn012:34206] regionserver.HRegionServer(2344): reportForDuty to master=cn012.l42scl.hortonworks.com,34094,1476973981831 with port=34206, startcode=1476973982007 2016-10-20 14:33:03,674 DEBUG [main-EventThread] zookeeper.ZKUtil(365): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on existing znode=/2/rs/cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,674 INFO [M:0;cn012:34094] regionserver.HRegionServer(2344): reportForDuty to master=cn012.l42scl.hortonworks.com,34094,1476973981831 with port=34094, startcode=1476973981831 2016-10-20 14:33:03,674 DEBUG [cn012:34094.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(248): Starting the controller for procedure member:cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,675 DEBUG [M:0;cn012:34094] regionserver.HRegionServer(2363): Master is not running yet 2016-10-20 14:33:03,675 WARN [M:0;cn012:34094] regionserver.HRegionServer(941): reportForDuty failed; sleeping and then retrying. 2016-10-20 14:33:03,677 DEBUG [main-EventThread] zookeeper.ZKUtil(365): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on existing znode=/2/rs/cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:03,679 DEBUG [AsyncRpcChannel-pool2-t4] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2016-10-20 14:33:03,681 DEBUG [RpcServer.listener,port=34094] ipc.RpcServer$Listener(880): RpcServer.listener,port=34094: connection from 172.18.128.12:41575; # active connections: 1 2016-10-20 14:33:03,681 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /2/rs/cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,681 INFO [cn012:34094.activeMasterManager] master.MasterCoprocessorHost(91): System coprocessor loading is enabled 2016-10-20 14:33:03,682 INFO [cn012:34094.activeMasterManager] coprocessor.CoprocessorHost(161): System coprocessor org.apache.hadoop.hbase.backup.master.BackupController was loaded successfully with priority (536870911). 2016-10-20 14:33:03,683 DEBUG [cn012:34094.activeMasterManager] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-cn012:34094, corePoolSize=5, maxPoolSize=5 2016-10-20 14:33:03,683 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=34094] ipc.RpcServer$Connection(1710): Auth successful for hbase.hfs.1 (auth:SIMPLE) 2016-10-20 14:33:03,683 DEBUG [cn012:34094.activeMasterManager] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-cn012:34094, corePoolSize=5, maxPoolSize=5 2016-10-20 14:33:03,684 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /2/rs/cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:03,684 DEBUG [cn012:34094.activeMasterManager] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-cn012:34094, corePoolSize=5, maxPoolSize=5 2016-10-20 14:33:03,684 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=34094] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 41575 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:03,684 DEBUG [cn012:34094.activeMasterManager] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-cn012:34094, corePoolSize=5, maxPoolSize=5 2016-10-20 14:33:03,685 DEBUG [cn012:34094.activeMasterManager] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-cn012:34094, corePoolSize=10, maxPoolSize=10 2016-10-20 14:33:03,686 DEBUG [cn012:34094.activeMasterManager] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-cn012:34094, corePoolSize=1, maxPoolSize=1 2016-10-20 14:33:03,686 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=34094] ipc.CallRunner(112): B.defaultRpcServer.handler=0,queue=0,port=34094: callId: 0 service: RegionServerStatusService methodName: RegionServerStartup size: 45 connection: 172.18.128.12:41575 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2301) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:264) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2229) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:136) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:111) at java.lang.Thread.run(Thread.java:745) 2016-10-20 14:33:03,686 INFO [cn012:34094.activeMasterManager] procedure2.ProcedureExecutor(487): Starting procedure executor threads=25 2016-10-20 14:33:03,687 INFO [cn012:34094.activeMasterManager] wal.WALProcedureStore(296): Starting WAL Procedure Store lease recovery 2016-10-20 14:33:03,688 WARN [cn012:34094.activeMasterManager] wal.WALProcedureStore(941): Log directory not found: File hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/MasterProcWALs does not exist. 2016-10-20 14:33:03,690 DEBUG [RS:0;cn012:34206] regionserver.HRegionServer(2363): Master is not running yet 2016-10-20 14:33:03,690 WARN [RS:0;cn012:34206] regionserver.HRegionServer(941): reportForDuty failed; sleeping and then retrying. 2016-10-20 14:33:03,692 DEBUG [cn012:34094.activeMasterManager] wal.WALProcedureStore(833): Roll new state log: 1 2016-10-20 14:33:03,693 INFO [cn012:34094.activeMasterManager] wal.WALProcedureStore(319): Lease acquired for flushLogId: 1 2016-10-20 14:33:03,693 DEBUG [cn012:34094.activeMasterManager] wal.WALProcedureStore(336): No state logs to replay. 2016-10-20 14:33:03,693 DEBUG [cn012:34094.activeMasterManager] procedure2.ProcedureExecutor$1(298): load procedures maxProcId=0 2016-10-20 14:33:03,698 DEBUG [cn012:34094.activeMasterManager] cleaner.CleanerChore(99): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2016-10-20 14:33:03,698 INFO [cn012:34094.activeMasterManager] zookeeper.RecoverableZooKeeper(120): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:33:03,705 DEBUG [cn012:34094.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(590): replicationLogCleaner0x0, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:33:03,707 DEBUG [cn012:34094.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(674): replicationLogCleaner-0x157e2823ec6000b connected 2016-10-20 14:33:03,730 DEBUG [cn012:34094.activeMasterManager] cleaner.CleanerChore(99): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2016-10-20 14:33:03,731 INFO [cn012:34094.activeMasterManager] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x56e7f655 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:33:03,738 DEBUG [cn012:34094.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x56e7f6550x0, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:33:03,740 DEBUG [cn012:34094.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x56e7f655-0x157e2823ec6000c connected 2016-10-20 14:33:03,740 DEBUG [cn012:34094.activeMasterManager] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@122de9bb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:33:03,741 DEBUG [cn012:34094.activeMasterManager] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:33:03,741 DEBUG [cn012:34094.activeMasterManager] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:33:03,741 DEBUG [cn012:34094.activeMasterManager] cleaner.CleanerChore(99): initialize cleaner=org.apache.hadoop.hbase.backup.master.BackupLogCleaner 2016-10-20 14:33:03,741 DEBUG [cn012:34094.activeMasterManager] cleaner.CleanerChore(99): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2016-10-20 14:33:03,742 DEBUG [cn012:34094.activeMasterManager] cleaner.CleanerChore(99): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2016-10-20 14:33:03,742 DEBUG [cn012:34094.activeMasterManager] cleaner.CleanerChore(99): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2016-10-20 14:33:03,742 INFO [cn012:34094.activeMasterManager] master.ServerManager(1008): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 1, maximum of 1, timeout of 4500 ms, interval of 1500 ms. 2016-10-20 14:33:03,743 INFO [M:0;cn012:34094] regionserver.HRegionServer(2344): reportForDuty to master=cn012.l42scl.hortonworks.com,34094,1476973981831 with port=34094, startcode=1476973981831 2016-10-20 14:33:03,744 INFO [M:0;cn012:34094] master.ServerManager(456): Registering server=cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,744 INFO [M:0;cn012:34094] regionserver.HRegionServer(1390): Config from master: hbase.rootdir=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998 2016-10-20 14:33:03,744 INFO [M:0;cn012:34094] regionserver.HRegionServer(1390): Config from master: fs.defaultFS=hdfs://localhost:46268 2016-10-20 14:33:03,744 INFO [M:0;cn012:34094] regionserver.HRegionServer(1390): Config from master: hbase.master.info.port=-1 2016-10-20 14:33:03,744 WARN [M:0;cn012:34094] hbase.ZNodeClearer(61): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2016-10-20 14:33:03,744 INFO [M:0;cn012:34094] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:03,745 DEBUG [M:0;cn012:34094] regionserver.HRegionServer(1654): logdir=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,766 DEBUG [M:0;cn012:34094] regionserver.Replication(152): ReplicationStatisticsThread 300 2016-10-20 14:33:03,766 INFO [M:0;cn012:34094] wal.WALFactory(144): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.RegionGroupingProvider 2016-10-20 14:33:03,766 INFO [M:0;cn012:34094] wal.RegionGroupingProvider(107): Instantiating RegionGroupingStrategy of type class org.apache.hadoop.hbase.wal.BoundedGroupingStrategy 2016-10-20 14:33:03,766 INFO [M:0;cn012:34094] regionserver.MetricsRegionServerWrapperImpl(139): Computing regionserver metrics every 5000 milliseconds 2016-10-20 14:33:03,771 DEBUG [M:0;cn012:34094] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-cn012:34094, corePoolSize=3, maxPoolSize=3 2016-10-20 14:33:03,771 DEBUG [M:0;cn012:34094] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-cn012:34094, corePoolSize=1, maxPoolSize=1 2016-10-20 14:33:03,771 DEBUG [M:0;cn012:34094] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-cn012:34094, corePoolSize=3, maxPoolSize=3 2016-10-20 14:33:03,772 DEBUG [M:0;cn012:34094] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-cn012:34094, corePoolSize=1, maxPoolSize=1 2016-10-20 14:33:03,772 DEBUG [M:0;cn012:34094] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-cn012:34094, corePoolSize=2, maxPoolSize=2 2016-10-20 14:33:03,772 DEBUG [M:0;cn012:34094] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-cn012:34094, corePoolSize=10, maxPoolSize=10 2016-10-20 14:33:03,772 DEBUG [M:0;cn012:34094] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-cn012:34094, corePoolSize=3, maxPoolSize=3 2016-10-20 14:33:03,774 DEBUG [M:0;cn012:34094] zookeeper.ZKUtil(365): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on existing znode=/2/rs/cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,775 DEBUG [M:0;cn012:34094] zookeeper.ZKUtil(365): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on existing znode=/2/rs/cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:03,775 INFO [M:0;cn012:34094] regionserver.ReplicationSourceManager(246): Current list of replicators: [cn012.l42scl.hortonworks.com,34094,1476973981831] other RSs: [cn012.l42scl.hortonworks.com,34094,1476973981831, cn012.l42scl.hortonworks.com,34206,1476973982007] 2016-10-20 14:33:03,793 INFO [cn012:34094.activeMasterManager] master.ServerManager(1025): Finished waiting for region servers count to settle; checked in 1, slept for 51 ms, expecting minimum of 1, maximum of 1, master is running 2016-10-20 14:33:03,794 INFO [cn012:34094.activeMasterManager] master.ServerManager(456): Registering server=cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:03,794 INFO [cn012:34094.activeMasterManager] master.HMaster(714): Registered server found up in zk but who has not yet reported in: cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:03,796 DEBUG [cn012:34094.activeMasterManager] zookeeper.ZKUtil(624): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Unable to get data of znode /2/meta-region-server because node does not exist (not an error) 2016-10-20 14:33:03,823 INFO [M:0;cn012:34094] regionserver.HeapMemoryManager(191): Starting HeapMemoryTuner chore. 2016-10-20 14:33:03,823 INFO [SplitLogWorker-cn012:34094] regionserver.SplitLogWorker(134): SplitLogWorker cn012.l42scl.hortonworks.com,34094,1476973981831 starting 2016-10-20 14:33:03,823 INFO [M:0;cn012:34094] regionserver.HRegionServer(1412): Serving as cn012.l42scl.hortonworks.com,34094,1476973981831, RpcServer on cn012.l42scl.hortonworks.com/172.18.128.12:34094, sessionid=0x157e2823ec60007 2016-10-20 14:33:03,823 DEBUG [M:0;cn012:34094] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc is starting 2016-10-20 14:33:03,824 DEBUG [M:0;cn012:34094] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,824 DEBUG [M:0;cn012:34094] procedure.ZKProcedureMemberRpcs(356): Starting procedure member 'cn012.l42scl.hortonworks.com,34094,1476973981831' 2016-10-20 14:33:03,824 DEBUG [M:0;cn012:34094] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/2/flush-table-proc/abort' 2016-10-20 14:33:03,825 DEBUG [M:0;cn012:34094] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/2/flush-table-proc/acquired' 2016-10-20 14:33:03,826 DEBUG [cn012:34094.activeMasterManager] zookeeper.ZKUtil(624): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Unable to get data of znode /2/meta-region-server because node does not exist (not an error) 2016-10-20 14:33:03,826 DEBUG [M:0;cn012:34094] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc is started 2016-10-20 14:33:03,826 DEBUG [M:0;cn012:34094] procedure.RegionServerProcedureManagerHost(51): Procedure backup-proc is starting 2016-10-20 14:33:03,826 DEBUG [M:0;cn012:34094] procedure.ZKProcedureMemberRpcs(356): Starting procedure member 'cn012.l42scl.hortonworks.com,34094,1476973981831' 2016-10-20 14:33:03,826 DEBUG [M:0;cn012:34094] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/2/rolllog-proc/abort' 2016-10-20 14:33:03,826 INFO [cn012:34094.activeMasterManager] master.HMaster(942): Re-assigning hbase:meta with replicaId, 0 it was on null 2016-10-20 14:33:03,826 DEBUG [cn012:34094.activeMasterManager] master.AssignmentManager(1291): No previous transition plan found (or ignoring an existing plan) for hbase:meta,,1.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=cn012.l42scl.hortonworks.com,34094,1476973981831; 2 (online=2) available servers, forceNewPlan=false 2016-10-20 14:33:03,826 INFO [cn012:34094.activeMasterManager] master.AssignmentManager(1080): Assigning hbase:meta,,1.1588230740 to cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,827 INFO [cn012:34094.activeMasterManager] master.RegionStates(1106): Transition {1588230740 state=OFFLINE, ts=1476973983826, server=null} to {1588230740 state=PENDING_OPEN, ts=1476973983827, server=cn012.l42scl.hortonworks.com,34094,1476973981831} 2016-10-20 14:33:03,827 DEBUG [M:0;cn012:34094] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/2/rolllog-proc/acquired' 2016-10-20 14:33:03,827 INFO [cn012:34094.activeMasterManager] zookeeper.MetaTableLocator(439): Setting hbase:meta region location in ZooKeeper as cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,827 INFO [M:0;cn012:34094] regionserver.LogRollRegionServerProcedureManager(85): Started region server backup manager. 2016-10-20 14:33:03,827 DEBUG [M:0;cn012:34094] procedure.RegionServerProcedureManagerHost(53): Procedure backup-proc is started 2016-10-20 14:33:03,827 DEBUG [M:0;cn012:34094] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot is starting 2016-10-20 14:33:03,827 DEBUG [M:0;cn012:34094] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,828 DEBUG [M:0;cn012:34094] procedure.ZKProcedureMemberRpcs(356): Starting procedure member 'cn012.l42scl.hortonworks.com,34094,1476973981831' 2016-10-20 14:33:03,828 DEBUG [M:0;cn012:34094] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/2/online-snapshot/abort' 2016-10-20 14:33:03,829 DEBUG [cn012:34094.activeMasterManager] zookeeper.MetaTableLocator(451): META region location doesn't exist, create it 2016-10-20 14:33:03,829 DEBUG [M:0;cn012:34094] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/2/online-snapshot/acquired' 2016-10-20 14:33:03,838 DEBUG [M:0;cn012:34094] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot is started 2016-10-20 14:33:03,838 INFO [M:0;cn012:34094] quotas.RegionServerQuotaManager(62): Quota support disabled 2016-10-20 14:33:03,838 DEBUG [cn012:34094.activeMasterManager] master.ServerManager(934): New admin connection to cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,838 INFO [cn012:34094.activeMasterManager] regionserver.RSRpcServices(1666): Open hbase:meta,,1.1588230740 2016-10-20 14:33:03,839 INFO [RS_OPEN_META-cn012:34094-0] wal.WALFactory(144): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2016-10-20 14:33:03,840 DEBUG [cn012:34094.activeMasterManager] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476973983839}]}} 2016-10-20 14:33:03,846 INFO [RS_OPEN_META-cn012:34094-0] wal.FSHLog(530): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=cn012.l42scl.hortonworks.com%2C34094%2C1476973981831.meta, suffix=.meta, logDir=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34094,1476973981831, archiveDir=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/oldWALs 2016-10-20 14:33:03,864 INFO [RS_OPEN_META-cn012:34094-0] wal.FSHLog(1432): Slow sync cost: 12 ms, current pipeline: [] 2016-10-20 14:33:03,865 INFO [RS_OPEN_META-cn012:34094-0] wal.FSHLog(888): New WAL /user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34094,1476973981831/cn012.l42scl.hortonworks.com%2C34094%2C1476973981831.meta.1476973983846.meta 2016-10-20 14:33:03,866 DEBUG [RS_OPEN_META-cn012:34094-0] regionserver.HRegion(6339): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2016-10-20 14:33:03,867 DEBUG [RS_OPEN_META-cn012:34094-0] coprocessor.CoprocessorHost(181): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2016-10-20 14:33:03,867 DEBUG [RS_OPEN_META-cn012:34094-0] regionserver.HRegion(7445): Registered coprocessor service: region=hbase:meta,,1 service=hbase.pb.MultiRowMutationService 2016-10-20 14:33:03,868 INFO [RS_OPEN_META-cn012:34094-0] regionserver.RegionCoprocessorHost(376): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2016-10-20 14:33:03,868 DEBUG [RS_OPEN_META-cn012:34094-0] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table meta 1588230740 2016-10-20 14:33:03,869 DEBUG [RS_OPEN_META-cn012:34094-0] regionserver.HRegion(736): Instantiated hbase:meta,,1.1588230740 2016-10-20 14:33:03,873 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:03,873 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:33:03,875 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/info 2016-10-20 14:33:03,876 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:03,877 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:33:03,878 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/table 2016-10-20 14:33:03,881 DEBUG [RS_OPEN_META-cn012:34094-0] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740 2016-10-20 14:33:03,885 DEBUG [RS_OPEN_META-cn012:34094-0] regionserver.FlushLargeStoresPolicy(72): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table hbase:meta, use config (67108864) instead 2016-10-20 14:33:03,891 DEBUG [RS_OPEN_META-cn012:34094-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/recovered.edits/3.seqid to file, newSeqId=3, maxSeqId=2 2016-10-20 14:33:03,892 INFO [RS_OPEN_META-cn012:34094-0] regionserver.HRegion(871): Onlined 1588230740; next sequenceid=3 2016-10-20 14:33:03,896 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1957): Post open deploy tasks for hbase:meta,,1.1588230740 2016-10-20 14:33:03,896 DEBUG [PostOpenDeployTasks:1588230740] master.AssignmentManager(2884): Got transition OPENED for {1588230740 state=PENDING_OPEN, ts=1476973983827, server=cn012.l42scl.hortonworks.com,34094,1476973981831} from cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,897 INFO [PostOpenDeployTasks:1588230740] master.RegionStates(1106): Transition {1588230740 state=PENDING_OPEN, ts=1476973983827, server=cn012.l42scl.hortonworks.com,34094,1476973981831} to {1588230740 state=OPEN, ts=1476973983897, server=cn012.l42scl.hortonworks.com,34094,1476973981831} 2016-10-20 14:33:03,897 INFO [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(439): Setting hbase:meta region location in ZooKeeper as cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,904 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/meta-region-server 2016-10-20 14:33:03,904 DEBUG [PostOpenDeployTasks:1588230740] master.RegionStates(452): Onlined 1588230740 on cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:03,905 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1984): Finished post open deploy task for hbase:meta,,1.1588230740 2016-10-20 14:33:03,907 DEBUG [RS_OPEN_META-cn012:34094-0] handler.OpenRegionHandler(126): Opened hbase:meta,,1.1588230740 on cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:04,047 INFO [cn012:34094.activeMasterManager] hbase.MetaTableAccessor(1700): Updated table hbase:meta state to ENABLED in META 2016-10-20 14:33:04,048 DEBUG [cn012:34094.activeMasterManager] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476973984048}]}} 2016-10-20 14:33:04,053 INFO [cn012:34094.activeMasterManager] hbase.MetaTableAccessor(1700): Updated table hbase:meta state to ENABLED in META 2016-10-20 14:33:04,063 DEBUG [cn012:34094.activeMasterManager] procedure.MasterProcedureScheduler(387): Wake event ProcedureEvent(server crash processing) 2016-10-20 14:33:04,063 INFO [cn012:34094.activeMasterManager] master.ServerManager(683): AssignmentManager hasn't finished failover cleanup; waiting 2016-10-20 14:33:04,066 INFO [cn012:34094.activeMasterManager] master.HMaster(969): hbase:meta with replicaId 0 assigned=1, location=cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:04,082 INFO [cn012:34094.activeMasterManager] master.AssignmentManager(555): Clean cluster startup. Don't reassign user regions 2016-10-20 14:33:04,093 INFO [cn012:34094.activeMasterManager] master.AssignmentManager(425): Joined the cluster in 25ms, failover=false 2016-10-20 14:33:04,096 DEBUG [cn012:34094.activeMasterManager] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/info 2016-10-20 14:33:04,098 DEBUG [cn012:34094.activeMasterManager] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/table 2016-10-20 14:33:04,099 INFO [cn012:34094.activeMasterManager] master.TableNamespaceManager(93): Namespace table not found. Creating... 2016-10-20 14:33:04,289 DEBUG [cn012:34094.activeMasterManager] procedure2.ProcedureExecutor(669): Procedure CreateTableProcedure (table=hbase:namespace) id=1 owner=hbase state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2016-10-20 14:33:04,324 DEBUG [ProcedureExecutor-0] lock.ZKInterProcessLockBase(226): Acquired a lock for /2/table-lock/hbase:namespace/write-master:340940000000000 2016-10-20 14:33:04,508 INFO [IPC Server handler 0 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-651c82c6-88ce-4f3b-bae6-e3460c66df10:NORMAL:127.0.0.1:43453|FINALIZED]]} size 0 2016-10-20 14:33:04,513 DEBUG [ProcedureExecutor-0] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2016-10-20 14:33:04,518 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6162): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/.tmp Table name == hbase:namespace 2016-10-20 14:33:04,540 INFO [IPC Server handler 2 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741832_1008{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a82af92a-d58e-4076-affe-976f93711253:NORMAL:127.0.0.1:43453|FINALIZED]]} size 0 2016-10-20 14:33:04,542 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(736): Instantiated hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f. 2016-10-20 14:33:04,543 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1419): Closing hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f.: disabling compactions & flushes 2016-10-20 14:33:04,544 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1446): Updates disabled for region hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f. 2016-10-20 14:33:04,544 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1552): Closed hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f. 2016-10-20 14:33:04,691 INFO [RS:0;cn012:34206] regionserver.HRegionServer(2344): reportForDuty to master=cn012.l42scl.hortonworks.com,34094,1476973981831 with port=34206, startcode=1476973982007 2016-10-20 14:33:04,699 INFO [B.defaultRpcServer.handler=1,queue=0,port=34094] master.ServerManager(456): Registering server=cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:04,700 INFO [RS:0;cn012:34206] regionserver.HRegionServer(1390): Config from master: hbase.rootdir=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998 2016-10-20 14:33:04,702 INFO [RS:0;cn012:34206] regionserver.HRegionServer(1390): Config from master: fs.defaultFS=hdfs://localhost:46268 2016-10-20 14:33:04,702 INFO [RS:0;cn012:34206] regionserver.HRegionServer(1390): Config from master: hbase.master.info.port=-1 2016-10-20 14:33:04,702 WARN [RS:0;cn012:34206] hbase.ZNodeClearer(61): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2016-10-20 14:33:04,702 INFO [RS:0;cn012:34206] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:04,703 DEBUG [RS:0;cn012:34206] regionserver.HRegionServer(1654): logdir=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:04,704 DEBUG [ProcedureExecutor-0] hbase.MetaTableAccessor(1374): Put{"totalColumns":1,"row":"hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":9223372036854775807}]}} 2016-10-20 14:33:04,715 INFO [ProcedureExecutor-0] hbase.MetaTableAccessor(1571): Added 1 2016-10-20 14:33:04,734 DEBUG [RS:0;cn012:34206] regionserver.Replication(152): ReplicationStatisticsThread 300 2016-10-20 14:33:04,735 INFO [RS:0;cn012:34206] wal.WALFactory(144): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.RegionGroupingProvider 2016-10-20 14:33:04,735 INFO [RS:0;cn012:34206] wal.RegionGroupingProvider(107): Instantiating RegionGroupingStrategy of type class org.apache.hadoop.hbase.wal.BoundedGroupingStrategy 2016-10-20 14:33:04,735 INFO [RS:0;cn012:34206] regionserver.MetricsRegionServerWrapperImpl(139): Computing regionserver metrics every 5000 milliseconds 2016-10-20 14:33:04,740 DEBUG [RS:0;cn012:34206] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-cn012:34206, corePoolSize=3, maxPoolSize=3 2016-10-20 14:33:04,740 DEBUG [RS:0;cn012:34206] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-cn012:34206, corePoolSize=1, maxPoolSize=1 2016-10-20 14:33:04,740 DEBUG [RS:0;cn012:34206] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-cn012:34206, corePoolSize=3, maxPoolSize=3 2016-10-20 14:33:04,740 DEBUG [RS:0;cn012:34206] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-cn012:34206, corePoolSize=1, maxPoolSize=1 2016-10-20 14:33:04,741 DEBUG [RS:0;cn012:34206] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-cn012:34206, corePoolSize=2, maxPoolSize=2 2016-10-20 14:33:04,741 DEBUG [RS:0;cn012:34206] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-cn012:34206, corePoolSize=10, maxPoolSize=10 2016-10-20 14:33:04,741 DEBUG [RS:0;cn012:34206] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-cn012:34206, corePoolSize=3, maxPoolSize=3 2016-10-20 14:33:04,745 DEBUG [RS:0;cn012:34206] zookeeper.ZKUtil(365): regionserver:34206-0x157e2823ec60008, quorum=localhost:56750, baseZNode=/2 Set watcher on existing znode=/2/rs/cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:04,746 DEBUG [RS:0;cn012:34206] zookeeper.ZKUtil(365): regionserver:34206-0x157e2823ec60008, quorum=localhost:56750, baseZNode=/2 Set watcher on existing znode=/2/rs/cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:04,746 INFO [RS:0;cn012:34206] regionserver.ReplicationSourceManager(246): Current list of replicators: [cn012.l42scl.hortonworks.com,34094,1476973981831, cn012.l42scl.hortonworks.com,34206,1476973982007] other RSs: [cn012.l42scl.hortonworks.com,34094,1476973981831, cn012.l42scl.hortonworks.com,34206,1476973982007] 2016-10-20 14:33:04,809 INFO [RS:0;cn012:34206] regionserver.HeapMemoryManager(191): Starting HeapMemoryTuner chore. 2016-10-20 14:33:04,809 INFO [SplitLogWorker-cn012:34206] regionserver.SplitLogWorker(134): SplitLogWorker cn012.l42scl.hortonworks.com,34206,1476973982007 starting 2016-10-20 14:33:04,809 INFO [RS:0;cn012:34206] regionserver.HRegionServer(1412): Serving as cn012.l42scl.hortonworks.com,34206,1476973982007, RpcServer on cn012.l42scl.hortonworks.com/172.18.128.12:34206, sessionid=0x157e2823ec60008 2016-10-20 14:33:04,809 DEBUG [RS:0;cn012:34206] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc is starting 2016-10-20 14:33:04,809 DEBUG [RS:0;cn012:34206] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:04,809 DEBUG [RS:0;cn012:34206] procedure.ZKProcedureMemberRpcs(356): Starting procedure member 'cn012.l42scl.hortonworks.com,34206,1476973982007' 2016-10-20 14:33:04,809 DEBUG [RS:0;cn012:34206] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/2/flush-table-proc/abort' 2016-10-20 14:33:04,810 DEBUG [RS:0;cn012:34206] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/2/flush-table-proc/acquired' 2016-10-20 14:33:04,811 DEBUG [RS:0;cn012:34206] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc is started 2016-10-20 14:33:04,811 DEBUG [RS:0;cn012:34206] procedure.RegionServerProcedureManagerHost(51): Procedure backup-proc is starting 2016-10-20 14:33:04,811 DEBUG [RS:0;cn012:34206] procedure.ZKProcedureMemberRpcs(356): Starting procedure member 'cn012.l42scl.hortonworks.com,34206,1476973982007' 2016-10-20 14:33:04,811 DEBUG [RS:0;cn012:34206] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/2/rolllog-proc/abort' 2016-10-20 14:33:04,812 DEBUG [RS:0;cn012:34206] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/2/rolllog-proc/acquired' 2016-10-20 14:33:04,813 INFO [RS:0;cn012:34206] regionserver.LogRollRegionServerProcedureManager(85): Started region server backup manager. 2016-10-20 14:33:04,813 DEBUG [RS:0;cn012:34206] procedure.RegionServerProcedureManagerHost(53): Procedure backup-proc is started 2016-10-20 14:33:04,813 DEBUG [RS:0;cn012:34206] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot is starting 2016-10-20 14:33:04,813 DEBUG [RS:0;cn012:34206] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:04,813 DEBUG [RS:0;cn012:34206] procedure.ZKProcedureMemberRpcs(356): Starting procedure member 'cn012.l42scl.hortonworks.com,34206,1476973982007' 2016-10-20 14:33:04,813 DEBUG [RS:0;cn012:34206] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/2/online-snapshot/abort' 2016-10-20 14:33:04,814 DEBUG [RS:0;cn012:34206] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/2/online-snapshot/acquired' 2016-10-20 14:33:04,815 DEBUG [RS:0;cn012:34206] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot is started 2016-10-20 14:33:04,815 INFO [RS:0;cn012:34206] quotas.RegionServerQuotaManager(62): Quota support disabled 2016-10-20 14:33:04,847 INFO [M:0;cn012:34094] wal.FSHLog(530): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=cn012.l42scl.hortonworks.com%2C34094%2C1476973981831.regiongroup-0, suffix=, logDir=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34094,1476973981831, archiveDir=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/oldWALs 2016-10-20 14:33:04,865 INFO [M:0;cn012:34094] wal.FSHLog(1432): Slow sync cost: 12 ms, current pipeline: [] 2016-10-20 14:33:04,865 INFO [M:0;cn012:34094] wal.FSHLog(888): New WAL /user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34094,1476973981831/cn012.l42scl.hortonworks.com%2C34094%2C1476973981831.regiongroup-0.1476973984847 2016-10-20 14:33:04,869 INFO [ProcedureExecutor-0] master.AssignmentManager(726): Assigning 1 region(s) to cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:04,870 ERROR [ProcedureExecutor-0] master.TableStateManager(134): Unable to get table hbase:namespace state org.apache.hadoop.hbase.TableNotFoundException: hbase:namespace at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546) at org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494) 2016-10-20 14:33:04,872 INFO [ProcedureExecutor-0] master.RegionStates(1106): Transition {37fa3cc8df36677363de1aacf6de184f state=OFFLINE, ts=1476973984869, server=null} to {37fa3cc8df36677363de1aacf6de184f state=PENDING_OPEN, ts=1476973984872, server=cn012.l42scl.hortonworks.com,34094,1476973981831} 2016-10-20 14:33:04,872 INFO [ProcedureExecutor-0] master.RegionStateStore(207): Updating hbase:meta row hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f. with state=PENDING_OPEN, sn=cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:04,879 INFO [ProcedureExecutor-0] regionserver.RSRpcServices(1666): Open hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f. 2016-10-20 14:33:04,900 DEBUG [ProcedureExecutor-0] master.AssignmentManager(897): Bulk assigning done for cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:04,902 DEBUG [ProcedureExecutor-0] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476973984901}]}} 2016-10-20 14:33:04,905 INFO [ProcedureExecutor-0] hbase.MetaTableAccessor(1700): Updated table hbase:namespace state to ENABLED in META 2016-10-20 14:33:04,907 INFO [RS_OPEN_REGION-cn012:34094-0] wal.FSHLog(530): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=cn012.l42scl.hortonworks.com%2C34094%2C1476973981831.regiongroup-1, suffix=, logDir=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34094,1476973981831, archiveDir=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/oldWALs 2016-10-20 14:33:04,920 INFO [RS_OPEN_REGION-cn012:34094-0] wal.FSHLog(1432): Slow sync cost: 9 ms, current pipeline: [] 2016-10-20 14:33:04,923 INFO [RS_OPEN_REGION-cn012:34094-0] wal.FSHLog(888): New WAL /user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34094,1476973981831/cn012.l42scl.hortonworks.com%2C34094%2C1476973981831.regiongroup-1.1476973984907 2016-10-20 14:33:04,925 DEBUG [RS_OPEN_REGION-cn012:34094-0] regionserver.HRegion(6339): Opening region: {ENCODED => 37fa3cc8df36677363de1aacf6de184f, NAME => 'hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f.', STARTKEY => '', ENDKEY => ''} 2016-10-20 14:33:04,926 DEBUG [RS_OPEN_REGION-cn012:34094-0] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table namespace 37fa3cc8df36677363de1aacf6de184f 2016-10-20 14:33:04,928 DEBUG [RS_OPEN_REGION-cn012:34094-0] regionserver.HRegion(736): Instantiated hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f. 2016-10-20 14:33:04,932 INFO [StoreOpener-37fa3cc8df36677363de1aacf6de184f-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:04,933 INFO [StoreOpener-37fa3cc8df36677363de1aacf6de184f-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:33:04,935 DEBUG [StoreOpener-37fa3cc8df36677363de1aacf6de184f-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/namespace/37fa3cc8df36677363de1aacf6de184f/info 2016-10-20 14:33:04,936 DEBUG [RS_OPEN_REGION-cn012:34094-0] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/namespace/37fa3cc8df36677363de1aacf6de184f 2016-10-20 14:33:04,946 DEBUG [RS_OPEN_REGION-cn012:34094-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/namespace/37fa3cc8df36677363de1aacf6de184f/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:33:04,946 INFO [RS_OPEN_REGION-cn012:34094-0] regionserver.HRegion(871): Onlined 37fa3cc8df36677363de1aacf6de184f; next sequenceid=2 2016-10-20 14:33:04,950 INFO [PostOpenDeployTasks:37fa3cc8df36677363de1aacf6de184f] regionserver.HRegionServer(1957): Post open deploy tasks for hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f. 2016-10-20 14:33:04,950 DEBUG [PostOpenDeployTasks:37fa3cc8df36677363de1aacf6de184f] master.AssignmentManager(2884): Got transition OPENED for {37fa3cc8df36677363de1aacf6de184f state=PENDING_OPEN, ts=1476973984872, server=cn012.l42scl.hortonworks.com,34094,1476973981831} from cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:04,951 INFO [PostOpenDeployTasks:37fa3cc8df36677363de1aacf6de184f] master.RegionStates(1106): Transition {37fa3cc8df36677363de1aacf6de184f state=PENDING_OPEN, ts=1476973984872, server=cn012.l42scl.hortonworks.com,34094,1476973981831} to {37fa3cc8df36677363de1aacf6de184f state=OPEN, ts=1476973984950, server=cn012.l42scl.hortonworks.com,34094,1476973981831} 2016-10-20 14:33:04,951 INFO [PostOpenDeployTasks:37fa3cc8df36677363de1aacf6de184f] master.RegionStateStore(207): Updating hbase:meta row hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f. with state=OPEN, openSeqNum=2, server=cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:04,954 DEBUG [PostOpenDeployTasks:37fa3cc8df36677363de1aacf6de184f] master.RegionStates(452): Onlined 37fa3cc8df36677363de1aacf6de184f on cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:04,955 DEBUG [PostOpenDeployTasks:37fa3cc8df36677363de1aacf6de184f] regionserver.HRegionServer(1984): Finished post open deploy task for hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f. 2016-10-20 14:33:04,955 DEBUG [RS_OPEN_REGION-cn012:34094-0] handler.OpenRegionHandler(126): Opened hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f. on cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:33:05,001 DEBUG [cn012:34094.activeMasterManager] zookeeper.ZKUtil(367): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on znode that does not yet exist, /2/namespace 2016-10-20 14:33:05,013 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/2/namespace 2016-10-20 14:33:05,217 DEBUG [cn012:34094.activeMasterManager] procedure2.ProcedureExecutor(669): Procedure CreateNamespaceProcedure (Namespace=default) id=2 owner=hbase state=RUNNABLE:CREATE_NAMESPACE_PREPARE added to the store. 2016-10-20 14:33:05,379 DEBUG [ProcedureExecutor-0] lock.ZKInterProcessLockBase(328): Released /2/table-lock/hbase:namespace/write-master:340940000000000 2016-10-20 14:33:05,380 DEBUG [ProcedureExecutor-0] procedure2.ProcedureExecutor(870): Procedure completed in 1.1150sec: CreateTableProcedure (table=hbase:namespace) id=1 owner=hbase state=FINISHED 2016-10-20 14:33:05,825 INFO [RS:0;cn012:34206] wal.FSHLog(530): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=cn012.l42scl.hortonworks.com%2C34206%2C1476973982007.regiongroup-0, suffix=, logDir=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34206,1476973982007, archiveDir=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/oldWALs 2016-10-20 14:33:05,850 INFO [RS:0;cn012:34206] wal.FSHLog(1432): Slow sync cost: 20 ms, current pipeline: [] 2016-10-20 14:33:05,851 INFO [RS:0;cn012:34206] wal.FSHLog(888): New WAL /user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34206,1476973982007/cn012.l42scl.hortonworks.com%2C34206%2C1476973982007.regiongroup-0.1476973985825 2016-10-20 14:33:05,879 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/2/namespace 2016-10-20 14:33:05,890 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node default with data: \x0A\x07default 2016-10-20 14:33:06,192 DEBUG [ProcedureExecutor-0] procedure2.ProcedureExecutor(870): Procedure completed in 1.0250sec: CreateNamespaceProcedure (Namespace=default) id=2 owner=hbase state=FINISHED 2016-10-20 14:33:06,376 DEBUG [cn012:34094.activeMasterManager] procedure2.ProcedureExecutor(669): Procedure CreateNamespaceProcedure (Namespace=hbase) id=3 owner=hbase state=RUNNABLE:CREATE_NAMESPACE_PREPARE added to the store. 2016-10-20 14:33:06,879 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/2/namespace 2016-10-20 14:33:06,891 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node default with data: \x0A\x07default 2016-10-20 14:33:06,891 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node hbase with data: \x0A\x05hbase 2016-10-20 14:33:06,999 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2016-10-20 14:33:07,192 DEBUG [ProcedureExecutor-3] procedure2.ProcedureExecutor(870): Procedure completed in 840msec: CreateNamespaceProcedure (Namespace=hbase) id=3 owner=hbase state=FINISHED 2016-10-20 14:33:07,213 DEBUG [cn012:34094.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /2/namespace/default already exists 2016-10-20 14:33:07,221 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/namespace/default 2016-10-20 14:33:07,229 DEBUG [cn012:34094.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /2/namespace/hbase already exists 2016-10-20 14:33:07,238 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/2/namespace/hbase 2016-10-20 14:33:07,238 INFO [cn012:34094.activeMasterManager] master.HMaster(811): Master has completed initialization 2016-10-20 14:33:07,238 DEBUG [cn012:34094.activeMasterManager] procedure.MasterProcedureScheduler(387): Wake event ProcedureEvent(master initialized) 2016-10-20 14:33:07,239 INFO [cn012:34094.activeMasterManager] quotas.MasterQuotaManager(72): Quota support disabled 2016-10-20 14:33:07,240 INFO [cn012:34094.activeMasterManager] zookeeper.ZooKeeperWatcher(225): not a secure deployment, proceeding 2016-10-20 14:33:07,259 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x174e1b69 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:33:07,271 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x174e1b690x0, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:33:07,274 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x174e1b69-0x157e2823ec6000d connected 2016-10-20 14:33:07,274 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1046498a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:33:07,275 DEBUG [main] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:33:07,275 DEBUG [main] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:33:07,274 INFO [cn012:34094.activeMasterManager] master.HMaster(1501): Client=null/null create 'hbase:backup', {NAME => 'meta', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'session', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2016-10-20 14:33:07,284 DEBUG [AsyncRpcChannel-pool2-t5] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:33:07,286 DEBUG [RpcServer.listener,port=34094] ipc.RpcServer$Listener(880): RpcServer.listener,port=34094: connection from 172.18.128.12:33114; # active connections: 2 2016-10-20 14:33:07,290 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=34094] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:33:07,291 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=34094] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 33114 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:07,305 INFO [main] hbase.HBaseTestingUtility(1090): Minicluster is up 2016-10-20 14:33:07,305 INFO [main] hbase.HBaseTestingUtility(1264): The hbase.fs.tmp.dir is set to /user/hbase/hbase-staging 2016-10-20 14:33:07,306 INFO [main] hbase.HBaseTestingUtility(2454): Starting mini mapreduce cluster... 2016-10-20 14:33:07,306 INFO [main] hbase.HBaseTestingUtility(744): Setting test.cache.data to /mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/cache_data in system properties and HBase conf 2016-10-20 14:33:07,306 INFO [main] hbase.HBaseTestingUtility(744): Setting hadoop.tmp.dir to /mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/hadoop_tmp in system properties and HBase conf 2016-10-20 14:33:07,306 INFO [main] hbase.HBaseTestingUtility(744): Setting hadoop.log.dir to /mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/hadoop_logs in system properties and HBase conf 2016-10-20 14:33:07,307 INFO [main] hbase.HBaseTestingUtility(744): Setting mapreduce.cluster.local.dir to /mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/mapred_local in system properties and HBase conf 2016-10-20 14:33:07,307 INFO [main] hbase.HBaseTestingUtility(744): Setting mapreduce.cluster.temp.dir to /mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/mapred_temp in system properties and HBase conf 2016-10-20 14:33:07,307 INFO [main] hbase.HBaseTestingUtility(735): read short circuit is OFF 2016-10-20 14:33:07,377 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:07,413 INFO [IPC Server handler 7 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741840_1016{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:33:07,433 DEBUG [cn012:34094.activeMasterManager] procedure2.ProcedureExecutor(669): Procedure CreateTableProcedure (table=hbase:backup) id=4 owner=hbase state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2016-10-20 14:33:07,490 DEBUG [ProcedureExecutor-1] lock.ZKInterProcessLockBase(226): Acquired a lock for /2/table-lock/hbase:backup/write-master:340940000000000 2016-10-20 14:33:07,493 INFO [cn012:34094.activeMasterManager] master.BackupController(51): Created hbase:backup table 2016-10-20 14:33:07,662 INFO [IPC Server handler 2 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a82af92a-d58e-4076-affe-976f93711253:NORMAL:127.0.0.1:43453|FINALIZED]]} size 0 2016-10-20 14:33:07,666 DEBUG [ProcedureExecutor-1] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/.tmp/data/hbase/backup/.tabledesc/.tableinfo.0000000001 2016-10-20 14:33:07,669 INFO [RegionOpenAndInitThread-hbase:backup-1] regionserver.HRegion(6162): creating HRegion hbase:backup HTD == 'hbase:backup', {NAME => 'meta', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'session', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/.tmp Table name == hbase:backup 2016-10-20 14:33:07,685 INFO [IPC Server handler 9 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-651c82c6-88ce-4f3b-bae6-e3460c66df10:NORMAL:127.0.0.1:43453|RBW]]} size 0 2016-10-20 14:33:07,687 DEBUG [RegionOpenAndInitThread-hbase:backup-1] regionserver.HRegion(736): Instantiated hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5. 2016-10-20 14:33:07,688 DEBUG [RegionOpenAndInitThread-hbase:backup-1] regionserver.HRegion(1419): Closing hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5.: disabling compactions & flushes 2016-10-20 14:33:07,688 DEBUG [RegionOpenAndInitThread-hbase:backup-1] regionserver.HRegion(1446): Updates disabled for region hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5. 2016-10-20 14:33:07,688 INFO [RegionOpenAndInitThread-hbase:backup-1] regionserver.HRegion(1552): Closed hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5. 2016-10-20 14:33:07,844 DEBUG [ProcedureExecutor-1] hbase.MetaTableAccessor(1374): Put{"totalColumns":1,"row":"hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5.","families":{"info":[{"qualifier":"regioninfo","vlen":38,"tag":[],"timestamp":9223372036854775807}]}} 2016-10-20 14:33:07,848 INFO [ProcedureExecutor-1] hbase.MetaTableAccessor(1571): Added 1 2016-10-20 14:33:07,994 INFO [ProcedureExecutor-1] master.AssignmentManager(726): Assigning 1 region(s) to cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:07,997 ERROR [ProcedureExecutor-1] master.TableStateManager(134): Unable to get table hbase:backup state org.apache.hadoop.hbase.TableNotFoundException: hbase:backup at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546) at org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494) 2016-10-20 14:33:07,998 INFO [ProcedureExecutor-1] master.RegionStates(1106): Transition {4a6cefc1b47013b7aa8a0dd5a0a290e5 state=OFFLINE, ts=1476973987994, server=null} to {4a6cefc1b47013b7aa8a0dd5a0a290e5 state=PENDING_OPEN, ts=1476973987998, server=cn012.l42scl.hortonworks.com,34206,1476973982007} 2016-10-20 14:33:07,998 INFO [ProcedureExecutor-1] master.RegionStateStore(207): Updating hbase:meta row hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5. with state=PENDING_OPEN, sn=cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:08,001 DEBUG [ProcedureExecutor-1] master.ServerManager(934): New admin connection to cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:08,009 DEBUG [AsyncRpcChannel-pool2-t6] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service AdminService, sasl=false 2016-10-20 14:33:08,011 DEBUG [RpcServer.listener,port=34206] ipc.RpcServer$Listener(880): RpcServer.listener,port=34206: connection from 172.18.128.12:49270; # active connections: 1 2016-10-20 14:33:08,016 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=34206] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:33:08,017 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=34206] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 49270 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:08,018 INFO [PriorityRpcServer.handler=0,queue=0,port=34206] regionserver.RSRpcServices(1666): Open hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5. 2016-10-20 14:33:08,033 DEBUG [ProcedureExecutor-1] master.AssignmentManager(897): Bulk assigning done for cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:08,034 DEBUG [ProcedureExecutor-1] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"hbase:backup","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476973988033}]}} 2016-10-20 14:33:08,039 INFO [ProcedureExecutor-1] hbase.MetaTableAccessor(1700): Updated table hbase:backup state to ENABLED in META 2016-10-20 14:33:08,040 INFO [RS_OPEN_REGION-cn012:34206-0] wal.FSHLog(530): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=cn012.l42scl.hortonworks.com%2C34206%2C1476973982007.regiongroup-1, suffix=, logDir=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34206,1476973982007, archiveDir=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/oldWALs 2016-10-20 14:33:08,065 INFO [RS_OPEN_REGION-cn012:34206-0] wal.FSHLog(1432): Slow sync cost: 14 ms, current pipeline: [] 2016-10-20 14:33:08,069 INFO [RS_OPEN_REGION-cn012:34206-0] wal.FSHLog(888): New WAL /user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34206,1476973982007/cn012.l42scl.hortonworks.com%2C34206%2C1476973982007.regiongroup-1.1476973988041 2016-10-20 14:33:08,072 DEBUG [RS_OPEN_REGION-cn012:34206-0] regionserver.HRegion(6339): Opening region: {ENCODED => 4a6cefc1b47013b7aa8a0dd5a0a290e5, NAME => 'hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5.', STARTKEY => '', ENDKEY => ''} 2016-10-20 14:33:08,072 DEBUG [RS_OPEN_REGION-cn012:34206-0] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table backup 4a6cefc1b47013b7aa8a0dd5a0a290e5 2016-10-20 14:33:08,073 DEBUG [RS_OPEN_REGION-cn012:34206-0] regionserver.HRegion(736): Instantiated hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5. 2016-10-20 14:33:08,081 INFO [StoreOpener-4a6cefc1b47013b7aa8a0dd5a0a290e5-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:08,081 INFO [StoreOpener-4a6cefc1b47013b7aa8a0dd5a0a290e5-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:33:08,084 DEBUG [StoreOpener-4a6cefc1b47013b7aa8a0dd5a0a290e5-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/backup/4a6cefc1b47013b7aa8a0dd5a0a290e5/meta 2016-10-20 14:33:08,088 INFO [StoreOpener-4a6cefc1b47013b7aa8a0dd5a0a290e5-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:08,089 INFO [StoreOpener-4a6cefc1b47013b7aa8a0dd5a0a290e5-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:33:08,092 DEBUG [StoreOpener-4a6cefc1b47013b7aa8a0dd5a0a290e5-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/backup/4a6cefc1b47013b7aa8a0dd5a0a290e5/session 2016-10-20 14:33:08,093 DEBUG [RS_OPEN_REGION-cn012:34206-0] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/backup/4a6cefc1b47013b7aa8a0dd5a0a290e5 2016-10-20 14:33:08,098 DEBUG [RS_OPEN_REGION-cn012:34206-0] regionserver.FlushLargeStoresPolicy(72): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table hbase:backup, use config (67108864) instead 2016-10-20 14:33:08,105 DEBUG [RS_OPEN_REGION-cn012:34206-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/backup/4a6cefc1b47013b7aa8a0dd5a0a290e5/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:33:08,105 INFO [RS_OPEN_REGION-cn012:34206-0] regionserver.HRegion(871): Onlined 4a6cefc1b47013b7aa8a0dd5a0a290e5; next sequenceid=2 2016-10-20 14:33:08,111 INFO [PostOpenDeployTasks:4a6cefc1b47013b7aa8a0dd5a0a290e5] regionserver.HRegionServer(1957): Post open deploy tasks for hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5. 2016-10-20 14:33:08,114 DEBUG [PriorityRpcServer.handler=3,queue=1,port=34094] master.AssignmentManager(2884): Got transition OPENED for {4a6cefc1b47013b7aa8a0dd5a0a290e5 state=PENDING_OPEN, ts=1476973987998, server=cn012.l42scl.hortonworks.com,34206,1476973982007} from cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:08,114 INFO [PriorityRpcServer.handler=3,queue=1,port=34094] master.RegionStates(1106): Transition {4a6cefc1b47013b7aa8a0dd5a0a290e5 state=PENDING_OPEN, ts=1476973987998, server=cn012.l42scl.hortonworks.com,34206,1476973982007} to {4a6cefc1b47013b7aa8a0dd5a0a290e5 state=OPEN, ts=1476973988114, server=cn012.l42scl.hortonworks.com,34206,1476973982007} 2016-10-20 14:33:08,114 INFO [PriorityRpcServer.handler=3,queue=1,port=34094] master.RegionStateStore(207): Updating hbase:meta row hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5. with state=OPEN, openSeqNum=2, server=cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:08,118 DEBUG [PriorityRpcServer.handler=3,queue=1,port=34094] master.RegionStates(452): Onlined 4a6cefc1b47013b7aa8a0dd5a0a290e5 on cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:08,120 DEBUG [PostOpenDeployTasks:4a6cefc1b47013b7aa8a0dd5a0a290e5] regionserver.HRegionServer(1984): Finished post open deploy task for hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5. 2016-10-20 14:33:08,121 DEBUG [RS_OPEN_REGION-cn012:34206-0] handler.OpenRegionHandler(126): Opened hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5. on cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:33:08,237 WARN [main] containermanager.AuxServices(130): The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config. 2016-10-20 14:33:08,588 DEBUG [ProcedureExecutor-1] lock.ZKInterProcessLockBase(328): Released /2/table-lock/hbase:backup/write-master:340940000000000 2016-10-20 14:33:08,588 DEBUG [ProcedureExecutor-1] procedure2.ProcedureExecutor(870): Procedure completed in 1.1570sec: CreateTableProcedure (table=hbase:backup) id=4 owner=hbase state=FINISHED 2016-10-20 14:33:08,693 WARN [main] containermanager.AuxServices(130): The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config. 2016-10-20 14:33:13,592 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-nodemanager.properties,hadoop-metrics2.properties 2016-10-20 14:33:18,889 INFO [Thread-450] log.Slf4jLog(67): jetty-6.1.26 2016-10-20 14:33:18,898 INFO [Thread-450] log.Slf4jLog(67): Extract jar:file:/home/hbase/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.7.1/hadoop-yarn-common-2.7.1.jar!/webapps/jobhistory to /tmp/Jetty_cn012_l42scl_hortonworks_com_35858_jobhistory____.m870t0/webapp 2016-10-20 14:33:18,972 INFO [Thread-450] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@cn012.l42scl.hortonworks.com:35858 2016-10-20 14:33:20,567 INFO [RM-0] log.Slf4jLog(67): jetty-6.1.26 2016-10-20 14:33:20,572 INFO [RM-0] log.Slf4jLog(67): Extract jar:file:/home/hbase/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.7.1/hadoop-yarn-common-2.7.1.jar!/webapps/cluster to /tmp/Jetty_cn012_l42scl_hortonworks_com_36373_cluster____b3wkb2/webapp Oct 20, 2016 2:33:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices as a root resource class Oct 20, 2016 2:33:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.mapreduce.v2.hs.webapp.JAXBContextResolver as a provider class Oct 20, 2016 2:33:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class Oct 20, 2016 2:33:20 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM' Oct 20, 2016 2:33:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.mapreduce.v2.hs.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton" Oct 20, 2016 2:33:21 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton" Oct 20, 2016 2:33:21 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices to GuiceManagedComponentProvider with the scope "PerRequest" 2016-10-20 14:33:21,488 INFO [RM-0] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@cn012.l42scl.hortonworks.com:36373 Oct 20, 2016 2:33:21 PM com.google.inject.servlet.GuiceFilter setPipeline WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected. 2016-10-20 14:33:21,924 INFO [Thread-648] log.Slf4jLog(67): jetty-6.1.26 2016-10-20 14:33:21,931 INFO [Thread-648] log.Slf4jLog(67): Extract jar:file:/home/hbase/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.7.1/hadoop-yarn-common-2.7.1.jar!/webapps/node to /tmp/Jetty_cn012_l42scl_hortonworks_com_42067_node____l8pvf/webapp Oct 20, 2016 2:33:22 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.server.resourcemanager.webapp.JAXBContextResolver as a provider class Oct 20, 2016 2:33:22 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices as a root resource class Oct 20, 2016 2:33:22 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class Oct 20, 2016 2:33:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM' Oct 20, 2016 2:33:22 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.server.resourcemanager.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton" Oct 20, 2016 2:33:22 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton" Oct 20, 2016 2:33:22 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices to GuiceManagedComponentProvider with the scope "Singleton" 2016-10-20 14:33:22,291 INFO [Thread-648] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@cn012.l42scl.hortonworks.com:42067 Oct 20, 2016 2:33:22 PM com.google.inject.servlet.GuiceFilter setPipeline WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected. 2016-10-20 14:33:22,886 INFO [Thread-686] log.Slf4jLog(67): jetty-6.1.26 2016-10-20 14:33:22,893 INFO [Thread-686] log.Slf4jLog(67): Extract jar:file:/home/hbase/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.7.1/hadoop-yarn-common-2.7.1.jar!/webapps/node to /tmp/Jetty_cn012_l42scl_hortonworks_com_41399_node____.i66ima/webapp Oct 20, 2016 2:33:22 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebServices as a root resource class Oct 20, 2016 2:33:22 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class Oct 20, 2016 2:33:22 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.server.nodemanager.webapp.JAXBContextResolver as a provider class Oct 20, 2016 2:33:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM' Oct 20, 2016 2:33:22 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.server.nodemanager.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton" Oct 20, 2016 2:33:23 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton" Oct 20, 2016 2:33:23 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebServices to GuiceManagedComponentProvider with the scope "Singleton" 2016-10-20 14:33:23,099 INFO [Thread-686] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@cn012.l42scl.hortonworks.com:41399 Oct 20, 2016 2:33:23 PM com.google.inject.servlet.GuiceFilter setPipeline WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected. 2016-10-20 14:33:23,864 INFO [main] hbase.HBaseTestingUtility(2482): Mini mapreduce cluster started 2016-10-20 14:33:23,864 INFO [main] backup.TestBackupBase(116): ROOTDIR hdfs://localhost:37879/backupUT 2016-10-20 14:33:23,865 INFO [main] backup.TestBackupBase(118): REMOTE ROOTDIR hdfs://localhost:46268/backupUT 2016-10-20 14:33:23,890 DEBUG [main] client.ConnectionImplementation(604): Table hbase:backup should be available 2016-10-20 14:33:23,890 DEBUG [main] backup.TestBackupBase(140): backup table exists and available 2016-10-20 14:33:23,906 DEBUG [AsyncRpcChannel-pool2-t7] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service MasterService, sasl=false 2016-10-20 14:33:23,909 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42532; # active connections: 3 2016-10-20 14:33:23,912 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:33:23,913 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42532 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:23,926 INFO [B.defaultRpcServer.handler=1,queue=0,port=41289] master.HMaster(2497): Client=hbase//172.18.128.12 creating {NAME => 'ns1'} 2016-10-20 14:33:24,082 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure CreateNamespaceProcedure (Namespace=ns1) id=5 owner=hbase state=RUNNABLE:CREATE_NAMESPACE_PREPARE added to the store. 2016-10-20 14:33:24,102 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=5 2016-10-20 14:33:24,208 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=5 2016-10-20 14:33:24,411 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=5 2016-10-20 14:33:24,562 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/namespace 2016-10-20 14:33:24,573 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node default with data: \x0A\x07default 2016-10-20 14:33:24,574 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node ns1 with data: \x0A\x03ns1 2016-10-20 14:33:24,574 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node hbase with data: \x0A\x05hbase 2016-10-20 14:33:24,713 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=5 2016-10-20 14:33:24,874 DEBUG [ProcedureExecutor-4] procedure2.ProcedureExecutor(870): Procedure completed in 795msec: CreateNamespaceProcedure (Namespace=ns1) id=5 owner=hbase state=FINISHED 2016-10-20 14:33:25,217 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=5 2016-10-20 14:33:25,220 INFO [B.defaultRpcServer.handler=2,queue=0,port=41289] master.HMaster(2497): Client=hbase//172.18.128.12 creating {NAME => 'ns2'} 2016-10-20 14:33:25,365 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure CreateNamespaceProcedure (Namespace=ns2) id=6 owner=hbase state=RUNNABLE:CREATE_NAMESPACE_PREPARE added to the store. 2016-10-20 14:33:25,373 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=6 2016-10-20 14:33:25,476 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=6 2016-10-20 14:33:25,679 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=6 2016-10-20 14:33:25,846 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/namespace 2016-10-20 14:33:25,857 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node default with data: \x0A\x07default 2016-10-20 14:33:25,858 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node ns2 with data: \x0A\x03ns2 2016-10-20 14:33:25,858 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node ns1 with data: \x0A\x03ns1 2016-10-20 14:33:25,858 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node hbase with data: \x0A\x05hbase 2016-10-20 14:33:25,982 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=6 2016-10-20 14:33:26,165 DEBUG [ProcedureExecutor-5] procedure2.ProcedureExecutor(870): Procedure completed in 795msec: CreateNamespaceProcedure (Namespace=ns2) id=6 owner=hbase state=FINISHED 2016-10-20 14:33:26,484 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=6 2016-10-20 14:33:26,487 INFO [B.defaultRpcServer.handler=3,queue=0,port=41289] master.HMaster(2497): Client=hbase//172.18.128.12 creating {NAME => 'ns3'} 2016-10-20 14:33:26,632 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure CreateNamespaceProcedure (Namespace=ns3) id=7 owner=hbase state=RUNNABLE:CREATE_NAMESPACE_PREPARE added to the store. 2016-10-20 14:33:26,640 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=7 2016-10-20 14:33:26,743 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=7 2016-10-20 14:33:26,945 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=7 2016-10-20 14:33:27,112 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/namespace 2016-10-20 14:33:27,125 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node default with data: \x0A\x07default 2016-10-20 14:33:27,125 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node ns2 with data: \x0A\x03ns2 2016-10-20 14:33:27,125 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node ns1 with data: \x0A\x03ns1 2016-10-20 14:33:27,125 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node hbase with data: \x0A\x05hbase 2016-10-20 14:33:27,125 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node ns3 with data: \x0A\x03ns3 2016-10-20 14:33:27,248 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=7 2016-10-20 14:33:27,432 DEBUG [ProcedureExecutor-6] procedure2.ProcedureExecutor(870): Procedure completed in 795msec: CreateNamespaceProcedure (Namespace=ns3) id=7 owner=hbase state=FINISHED 2016-10-20 14:33:27,751 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=7 2016-10-20 14:33:27,754 INFO [B.defaultRpcServer.handler=1,queue=0,port=41289] master.HMaster(2497): Client=hbase//172.18.128.12 creating {NAME => 'ns4'} 2016-10-20 14:33:27,915 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure CreateNamespaceProcedure (Namespace=ns4) id=8 owner=hbase state=RUNNABLE:CREATE_NAMESPACE_PREPARE added to the store. 2016-10-20 14:33:27,922 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=8 2016-10-20 14:33:28,025 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=8 2016-10-20 14:33:28,228 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=8 2016-10-20 14:33:28,396 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/namespace 2016-10-20 14:33:28,410 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node default with data: \x0A\x07default 2016-10-20 14:33:28,410 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node ns2 with data: \x0A\x03ns2 2016-10-20 14:33:28,410 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node ns1 with data: \x0A\x03ns1 2016-10-20 14:33:28,410 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node hbase with data: \x0A\x05hbase 2016-10-20 14:33:28,411 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node ns4 with data: \x0A\x03ns4 2016-10-20 14:33:28,411 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node ns3 with data: \x0A\x03ns3 2016-10-20 14:33:28,531 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=8 2016-10-20 14:33:28,742 DEBUG [ProcedureExecutor-7] procedure2.ProcedureExecutor(870): Procedure completed in 833msec: CreateNamespaceProcedure (Namespace=ns4) id=8 owner=hbase state=FINISHED 2016-10-20 14:33:29,034 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=8 2016-10-20 14:33:29,048 INFO [B.defaultRpcServer.handler=4,queue=0,port=41289] master.HMaster(1501): Client=hbase//172.18.128.12 create 'ns1:test-1476974003890', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2016-10-20 14:33:29,192 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure CreateTableProcedure (table=ns1:test-1476974003890) id=9 owner=hbase state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2016-10-20 14:33:29,206 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=9 2016-10-20 14:33:29,231 DEBUG [ProcedureExecutor-8] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/ns1:test-1476974003890/write-master:412890000000000 2016-10-20 14:33:29,310 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=9 2016-10-20 14:33:29,412 INFO [IPC Server handler 9 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741841_1017{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 294 2016-10-20 14:33:29,513 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=9 2016-10-20 14:33:29,816 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=9 2016-10-20 14:33:29,818 DEBUG [ProcedureExecutor-8] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/test-1476974003890/.tabledesc/.tableinfo.0000000001 2016-10-20 14:33:29,821 INFO [RegionOpenAndInitThread-ns1:test-1476974003890-1] regionserver.HRegion(6162): creating HRegion ns1:test-1476974003890 HTD == 'ns1:test-1476974003890', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp Table name == ns1:test-1476974003890 2016-10-20 14:33:29,841 INFO [IPC Server handler 9 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741842_1018{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:33:29,843 DEBUG [RegionOpenAndInitThread-ns1:test-1476974003890-1] regionserver.HRegion(736): Instantiated ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:29,845 DEBUG [RegionOpenAndInitThread-ns1:test-1476974003890-1] regionserver.HRegion(1419): Closing ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.: disabling compactions & flushes 2016-10-20 14:33:29,846 DEBUG [RegionOpenAndInitThread-ns1:test-1476974003890-1] regionserver.HRegion(1446): Updates disabled for region ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:29,846 INFO [RegionOpenAndInitThread-ns1:test-1476974003890-1] regionserver.HRegion(1552): Closed ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:29,994 DEBUG [ProcedureExecutor-8] hbase.MetaTableAccessor(1374): Put{"totalColumns":1,"row":"ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.","families":{"info":[{"qualifier":"regioninfo","vlen":48,"tag":[],"timestamp":9223372036854775807}]}} 2016-10-20 14:33:29,999 INFO [ProcedureExecutor-8] hbase.MetaTableAccessor(1571): Added 1 2016-10-20 14:33:30,144 INFO [ProcedureExecutor-8] master.AssignmentManager(726): Assigning 1 region(s) to cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:30,145 ERROR [ProcedureExecutor-8] master.TableStateManager(134): Unable to get table ns1:test-1476974003890 state org.apache.hadoop.hbase.TableNotFoundException: ns1:test-1476974003890 at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546) at org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494) 2016-10-20 14:33:30,146 INFO [ProcedureExecutor-8] master.RegionStates(1106): Transition {4a7f646a7c8fc84833ef454ba337a246 state=OFFLINE, ts=1476974010143, server=null} to {4a7f646a7c8fc84833ef454ba337a246 state=PENDING_OPEN, ts=1476974010146, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:33:30,146 INFO [ProcedureExecutor-8] master.RegionStateStore(207): Updating hbase:meta row ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. with state=PENDING_OPEN, sn=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:30,154 INFO [PriorityRpcServer.handler=2,queue=0,port=37560] regionserver.RSRpcServices(1666): Open ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:30,170 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(6339): Opening region: {ENCODED => 4a7f646a7c8fc84833ef454ba337a246, NAME => 'ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.', STARTKEY => '', ENDKEY => ''} 2016-10-20 14:33:30,170 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table test-1476974003890 4a7f646a7c8fc84833ef454ba337a246 2016-10-20 14:33:30,171 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(736): Instantiated ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:30,178 INFO [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:30,180 INFO [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:33:30,182 DEBUG [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f 2016-10-20 14:33:30,184 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246 2016-10-20 14:33:30,195 DEBUG [RS_OPEN_REGION-cn012:37560-1] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:33:30,195 INFO [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(871): Onlined 4a7f646a7c8fc84833ef454ba337a246; next sequenceid=2 2016-10-20 14:33:30,202 INFO [PostOpenDeployTasks:4a7f646a7c8fc84833ef454ba337a246] regionserver.HRegionServer(1957): Post open deploy tasks for ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:30,204 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.AssignmentManager(2884): Got transition OPENED for {4a7f646a7c8fc84833ef454ba337a246 state=PENDING_OPEN, ts=1476974010146, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:30,204 INFO [B.defaultRpcServer.handler=4,queue=0,port=41289] master.RegionStates(1106): Transition {4a7f646a7c8fc84833ef454ba337a246 state=PENDING_OPEN, ts=1476974010146, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {4a7f646a7c8fc84833ef454ba337a246 state=OPEN, ts=1476974010204, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:33:30,204 INFO [B.defaultRpcServer.handler=4,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. with state=OPEN, openSeqNum=2, server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:30,207 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.RegionStates(452): Onlined 4a7f646a7c8fc84833ef454ba337a246 on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:30,207 DEBUG [ProcedureExecutor-8] master.AssignmentManager(897): Bulk assigning done for cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:30,208 DEBUG [ProcedureExecutor-8] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"ns1:test-1476974003890","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476974010207}]}} 2016-10-20 14:33:30,208 ERROR [B.defaultRpcServer.handler=4,queue=0,port=41289] master.TableStateManager(134): Unable to get table ns1:test-1476974003890 state org.apache.hadoop.hbase.TableNotFoundException: ns1:test-1476974003890 at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionOpen(AssignmentManager.java:2311) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionTransition(AssignmentManager.java:2891) at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1350) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8623) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2229) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:136) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:111) at java.lang.Thread.run(Thread.java:745) 2016-10-20 14:33:30,210 DEBUG [PostOpenDeployTasks:4a7f646a7c8fc84833ef454ba337a246] regionserver.HRegionServer(1984): Finished post open deploy task for ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:30,212 DEBUG [RS_OPEN_REGION-cn012:37560-1] handler.OpenRegionHandler(126): Opened ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:30,212 INFO [ProcedureExecutor-8] hbase.MetaTableAccessor(1700): Updated table ns1:test-1476974003890 state to ENABLED in META 2016-10-20 14:33:30,319 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=9 2016-10-20 14:33:30,662 DEBUG [ProcedureExecutor-8] lock.ZKInterProcessLockBase(328): Released /1/table-lock/ns1:test-1476974003890/write-master:412890000000000 2016-10-20 14:33:30,663 DEBUG [ProcedureExecutor-8] procedure2.ProcedureExecutor(870): Procedure completed in 1.4590sec: CreateTableProcedure (table=ns1:test-1476974003890) id=9 owner=hbase state=FINISHED 2016-10-20 14:33:31,322 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=9 2016-10-20 14:33:31,322 INFO [main] client.HBaseAdmin$TableFuture(3337): Operation: CREATE, Table Name: ns1:test-1476974003890 completed 2016-10-20 14:33:31,324 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x38ed139b connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:33:31,338 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x38ed139b0x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:33:31,341 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x38ed139b-0x157e2823ec6000e connected 2016-10-20 14:33:31,341 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a5272be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:33:31,342 DEBUG [main] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:33:31,342 DEBUG [main] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:33:31,350 DEBUG [AsyncRpcChannel-pool2-t8] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:33:31,351 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42562; # active connections: 4 2016-10-20 14:33:31,354 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:33:31,354 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42562 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:31,367 DEBUG [AsyncRpcChannel-pool2-t9] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:33:31,370 DEBUG [RpcServer.listener,port=37560] ipc.RpcServer$Listener(880): RpcServer.listener,port=37560: connection from 172.18.128.12:41932; # active connections: 2 2016-10-20 14:33:31,373 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:33:31,373 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 41932 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:31,376 INFO [B.defaultRpcServer.handler=0,queue=0,port=37560] regionserver.HRegion(7705): writing data to region ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. with WAL disabled. Data may be lost in the event of a crash. 2016-10-20 14:33:31,604 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.HMaster(1501): Client=hbase//172.18.128.12 create 'ns2:test-14769740038901', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2016-10-20 14:33:31,750 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure CreateTableProcedure (table=ns2:test-14769740038901) id=10 owner=hbase state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2016-10-20 14:33:31,760 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=10 2016-10-20 14:33:31,789 DEBUG [ProcedureExecutor-9] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/ns2:test-14769740038901/write-master:412890000000000 2016-10-20 14:33:31,863 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=10 2016-10-20 14:33:31,992 INFO [IPC Server handler 3 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741843_1019{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:31,997 DEBUG [ProcedureExecutor-9] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns2/test-14769740038901/.tabledesc/.tableinfo.0000000001 2016-10-20 14:33:32,000 INFO [RegionOpenAndInitThread-ns2:test-14769740038901-1] regionserver.HRegion(6162): creating HRegion ns2:test-14769740038901 HTD == 'ns2:test-14769740038901', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp Table name == ns2:test-14769740038901 2016-10-20 14:33:32,018 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741844_1020{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:33:32,019 DEBUG [RegionOpenAndInitThread-ns2:test-14769740038901-1] regionserver.HRegion(736): Instantiated ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. 2016-10-20 14:33:32,022 DEBUG [RegionOpenAndInitThread-ns2:test-14769740038901-1] regionserver.HRegion(1419): Closing ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391.: disabling compactions & flushes 2016-10-20 14:33:32,022 DEBUG [RegionOpenAndInitThread-ns2:test-14769740038901-1] regionserver.HRegion(1446): Updates disabled for region ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. 2016-10-20 14:33:32,022 INFO [RegionOpenAndInitThread-ns2:test-14769740038901-1] regionserver.HRegion(1552): Closed ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. 2016-10-20 14:33:32,066 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=10 2016-10-20 14:33:32,177 DEBUG [ProcedureExecutor-9] hbase.MetaTableAccessor(1374): Put{"totalColumns":1,"row":"ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807}]}} 2016-10-20 14:33:32,182 INFO [ProcedureExecutor-9] hbase.MetaTableAccessor(1571): Added 1 2016-10-20 14:33:32,336 INFO [ProcedureExecutor-9] master.AssignmentManager(726): Assigning 1 region(s) to cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:32,337 ERROR [ProcedureExecutor-9] master.TableStateManager(134): Unable to get table ns2:test-14769740038901 state org.apache.hadoop.hbase.TableNotFoundException: ns2:test-14769740038901 at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546) at org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494) 2016-10-20 14:33:32,338 INFO [ProcedureExecutor-9] master.RegionStates(1106): Transition {20fb71680e16fdda348e8695b59a8391 state=OFFLINE, ts=1476974012336, server=null} to {20fb71680e16fdda348e8695b59a8391 state=PENDING_OPEN, ts=1476974012338, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:33:32,339 INFO [ProcedureExecutor-9] master.RegionStateStore(207): Updating hbase:meta row ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. with state=PENDING_OPEN, sn=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:32,342 INFO [PriorityRpcServer.handler=4,queue=0,port=37560] regionserver.RSRpcServices(1666): Open ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. 2016-10-20 14:33:32,354 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(6339): Opening region: {ENCODED => 20fb71680e16fdda348e8695b59a8391, NAME => 'ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391.', STARTKEY => '', ENDKEY => ''} 2016-10-20 14:33:32,356 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table test-14769740038901 20fb71680e16fdda348e8695b59a8391 2016-10-20 14:33:32,357 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(736): Instantiated ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. 2016-10-20 14:33:32,365 INFO [StoreOpener-20fb71680e16fdda348e8695b59a8391-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:32,367 INFO [StoreOpener-20fb71680e16fdda348e8695b59a8391-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:33:32,369 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=10 2016-10-20 14:33:32,369 DEBUG [StoreOpener-20fb71680e16fdda348e8695b59a8391-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/f 2016-10-20 14:33:32,371 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391 2016-10-20 14:33:32,381 DEBUG [RS_OPEN_REGION-cn012:37560-2] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:33:32,381 INFO [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(871): Onlined 20fb71680e16fdda348e8695b59a8391; next sequenceid=2 2016-10-20 14:33:32,384 INFO [PostOpenDeployTasks:20fb71680e16fdda348e8695b59a8391] regionserver.HRegionServer(1957): Post open deploy tasks for ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. 2016-10-20 14:33:32,386 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.AssignmentManager(2884): Got transition OPENED for {20fb71680e16fdda348e8695b59a8391 state=PENDING_OPEN, ts=1476974012338, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:32,386 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.RegionStates(1106): Transition {20fb71680e16fdda348e8695b59a8391 state=PENDING_OPEN, ts=1476974012338, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {20fb71680e16fdda348e8695b59a8391 state=OPEN, ts=1476974012386, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:33:32,386 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. with state=OPEN, openSeqNum=2, server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:32,389 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.RegionStates(452): Onlined 20fb71680e16fdda348e8695b59a8391 on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:32,390 DEBUG [ProcedureExecutor-9] master.AssignmentManager(897): Bulk assigning done for cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:32,390 DEBUG [ProcedureExecutor-9] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"ns2:test-14769740038901","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476974012390}]}} 2016-10-20 14:33:32,390 ERROR [B.defaultRpcServer.handler=0,queue=0,port=41289] master.TableStateManager(134): Unable to get table ns2:test-14769740038901 state org.apache.hadoop.hbase.TableNotFoundException: ns2:test-14769740038901 at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionOpen(AssignmentManager.java:2311) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionTransition(AssignmentManager.java:2891) at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1350) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8623) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2229) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:136) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:111) at java.lang.Thread.run(Thread.java:745) 2016-10-20 14:33:32,392 DEBUG [PostOpenDeployTasks:20fb71680e16fdda348e8695b59a8391] regionserver.HRegionServer(1984): Finished post open deploy task for ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. 2016-10-20 14:33:32,396 DEBUG [RS_OPEN_REGION-cn012:37560-2] handler.OpenRegionHandler(126): Opened ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:32,397 INFO [ProcedureExecutor-9] hbase.MetaTableAccessor(1700): Updated table ns2:test-14769740038901 state to ENABLED in META 2016-10-20 14:33:32,870 DEBUG [ProcedureExecutor-9] lock.ZKInterProcessLockBase(328): Released /1/table-lock/ns2:test-14769740038901/write-master:412890000000000 2016-10-20 14:33:32,871 DEBUG [ProcedureExecutor-9] procedure2.ProcedureExecutor(870): Procedure completed in 1.1040sec: CreateTableProcedure (table=ns2:test-14769740038901) id=10 owner=hbase state=FINISHED 2016-10-20 14:33:32,872 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=10 2016-10-20 14:33:32,873 INFO [main] client.HBaseAdmin$TableFuture(3337): Operation: CREATE, Table Name: ns2:test-14769740038901 completed 2016-10-20 14:33:32,884 INFO [B.defaultRpcServer.handler=4,queue=0,port=37560] regionserver.HRegion(7705): writing data to region ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. with WAL disabled. Data may be lost in the event of a crash. 2016-10-20 14:33:33,080 INFO [B.defaultRpcServer.handler=1,queue=0,port=41289] master.HMaster(1501): Client=hbase//172.18.128.12 create 'ns3:test-14769740038902', {NAME => 'f', BLOOMFILTER => 'NONE', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2016-10-20 14:33:33,233 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure CreateTableProcedure (table=ns3:test-14769740038902) id=11 owner=hbase state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2016-10-20 14:33:33,246 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=11 2016-10-20 14:33:33,272 DEBUG [ProcedureExecutor-10] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/ns3:test-14769740038902/write-master:412890000000000 2016-10-20 14:33:33,348 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=11 2016-10-20 14:33:33,455 INFO [IPC Server handler 1 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741845_1021{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:33:33,460 DEBUG [ProcedureExecutor-10] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns3/test-14769740038902/.tabledesc/.tableinfo.0000000001 2016-10-20 14:33:33,462 INFO [RegionOpenAndInitThread-ns3:test-14769740038902-1] regionserver.HRegion(6162): creating HRegion ns3:test-14769740038902 HTD == 'ns3:test-14769740038902', {NAME => 'f', BLOOMFILTER => 'NONE', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp Table name == ns3:test-14769740038902 2016-10-20 14:33:33,482 INFO [IPC Server handler 5 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741846_1022{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:33:33,484 DEBUG [RegionOpenAndInitThread-ns3:test-14769740038902-1] regionserver.HRegion(736): Instantiated ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e. 2016-10-20 14:33:33,487 DEBUG [RegionOpenAndInitThread-ns3:test-14769740038902-1] regionserver.HRegion(1419): Closing ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e.: disabling compactions & flushes 2016-10-20 14:33:33,487 DEBUG [RegionOpenAndInitThread-ns3:test-14769740038902-1] regionserver.HRegion(1446): Updates disabled for region ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e. 2016-10-20 14:33:33,487 INFO [RegionOpenAndInitThread-ns3:test-14769740038902-1] regionserver.HRegion(1552): Closed ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e. 2016-10-20 14:33:33,552 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=11 2016-10-20 14:33:33,677 DEBUG [ProcedureExecutor-10] hbase.MetaTableAccessor(1374): Put{"totalColumns":1,"row":"ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807}]}} 2016-10-20 14:33:33,681 INFO [ProcedureExecutor-10] hbase.MetaTableAccessor(1571): Added 1 2016-10-20 14:33:33,836 INFO [ProcedureExecutor-10] master.AssignmentManager(726): Assigning 1 region(s) to cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:33,837 ERROR [ProcedureExecutor-10] master.TableStateManager(134): Unable to get table ns3:test-14769740038902 state org.apache.hadoop.hbase.TableNotFoundException: ns3:test-14769740038902 at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546) at org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494) 2016-10-20 14:33:33,838 INFO [ProcedureExecutor-10] master.RegionStates(1106): Transition {0bcceb715ec0f6a2bed0aef31b8a540e state=OFFLINE, ts=1476974013836, server=null} to {0bcceb715ec0f6a2bed0aef31b8a540e state=PENDING_OPEN, ts=1476974013838, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:33:33,838 INFO [ProcedureExecutor-10] master.RegionStateStore(207): Updating hbase:meta row ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e. with state=PENDING_OPEN, sn=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:33,842 INFO [PriorityRpcServer.handler=1,queue=1,port=37560] regionserver.RSRpcServices(1666): Open ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e. 2016-10-20 14:33:33,854 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(6339): Opening region: {ENCODED => 0bcceb715ec0f6a2bed0aef31b8a540e, NAME => 'ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e.', STARTKEY => '', ENDKEY => ''} 2016-10-20 14:33:33,855 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table test-14769740038902 0bcceb715ec0f6a2bed0aef31b8a540e 2016-10-20 14:33:33,856 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(736): Instantiated ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e. 2016-10-20 14:33:33,856 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=11 2016-10-20 14:33:33,862 INFO [StoreOpener-0bcceb715ec0f6a2bed0aef31b8a540e-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:33,863 INFO [StoreOpener-0bcceb715ec0f6a2bed0aef31b8a540e-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:33:33,865 DEBUG [StoreOpener-0bcceb715ec0f6a2bed0aef31b8a540e-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns3/test-14769740038902/0bcceb715ec0f6a2bed0aef31b8a540e/f 2016-10-20 14:33:33,867 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns3/test-14769740038902/0bcceb715ec0f6a2bed0aef31b8a540e 2016-10-20 14:33:33,877 DEBUG [RS_OPEN_REGION-cn012:37560-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns3/test-14769740038902/0bcceb715ec0f6a2bed0aef31b8a540e/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:33:33,877 INFO [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(871): Onlined 0bcceb715ec0f6a2bed0aef31b8a540e; next sequenceid=2 2016-10-20 14:33:33,880 INFO [PostOpenDeployTasks:0bcceb715ec0f6a2bed0aef31b8a540e] regionserver.HRegionServer(1957): Post open deploy tasks for ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e. 2016-10-20 14:33:33,881 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.AssignmentManager(2884): Got transition OPENED for {0bcceb715ec0f6a2bed0aef31b8a540e state=PENDING_OPEN, ts=1476974013838, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:33,881 INFO [B.defaultRpcServer.handler=1,queue=0,port=41289] master.RegionStates(1106): Transition {0bcceb715ec0f6a2bed0aef31b8a540e state=PENDING_OPEN, ts=1476974013838, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {0bcceb715ec0f6a2bed0aef31b8a540e state=OPEN, ts=1476974013881, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:33:33,882 INFO [B.defaultRpcServer.handler=1,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e. with state=OPEN, openSeqNum=2, server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:33,884 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.RegionStates(452): Onlined 0bcceb715ec0f6a2bed0aef31b8a540e on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:33,884 DEBUG [ProcedureExecutor-10] master.AssignmentManager(897): Bulk assigning done for cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:33,885 DEBUG [ProcedureExecutor-10] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"ns3:test-14769740038902","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476974013884}]}} 2016-10-20 14:33:33,885 ERROR [B.defaultRpcServer.handler=1,queue=0,port=41289] master.TableStateManager(134): Unable to get table ns3:test-14769740038902 state org.apache.hadoop.hbase.TableNotFoundException: ns3:test-14769740038902 at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionOpen(AssignmentManager.java:2311) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionTransition(AssignmentManager.java:2891) at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1350) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8623) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2229) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:136) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:111) at java.lang.Thread.run(Thread.java:745) 2016-10-20 14:33:33,886 DEBUG [PostOpenDeployTasks:0bcceb715ec0f6a2bed0aef31b8a540e] regionserver.HRegionServer(1984): Finished post open deploy task for ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e. 2016-10-20 14:33:33,890 DEBUG [RS_OPEN_REGION-cn012:37560-0] handler.OpenRegionHandler(126): Opened ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e. on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:33,891 INFO [ProcedureExecutor-10] hbase.MetaTableAccessor(1700): Updated table ns3:test-14769740038902 state to ENABLED in META 2016-10-20 14:33:34,359 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=11 2016-10-20 14:33:34,362 DEBUG [ProcedureExecutor-10] lock.ZKInterProcessLockBase(328): Released /1/table-lock/ns3:test-14769740038902/write-master:412890000000000 2016-10-20 14:33:34,362 DEBUG [ProcedureExecutor-10] procedure2.ProcedureExecutor(870): Procedure completed in 1.1190sec: CreateTableProcedure (table=ns3:test-14769740038902) id=11 owner=hbase state=FINISHED 2016-10-20 14:33:35,362 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=11 2016-10-20 14:33:35,363 INFO [main] client.HBaseAdmin$TableFuture(3337): Operation: CREATE, Table Name: ns3:test-14769740038902 completed 2016-10-20 14:33:35,368 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2016-10-20 14:33:35,382 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2016-10-20 14:33:35,387 INFO [B.defaultRpcServer.handler=4,queue=0,port=41289] master.HMaster(1501): Client=hbase//172.18.128.12 create 'ns4:test-14769740038903', {NAME => 'f', BLOOMFILTER => 'NONE', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2016-10-20 14:33:35,540 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure CreateTableProcedure (table=ns4:test-14769740038903) id=12 owner=hbase state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2016-10-20 14:33:35,551 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=12 2016-10-20 14:33:35,580 DEBUG [ProcedureExecutor-11] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/ns4:test-14769740038903/write-master:412890000000000 2016-10-20 14:33:35,653 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=12 2016-10-20 14:33:35,728 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-jobhistoryserver.properties,hadoop-metrics2.properties 2016-10-20 14:33:35,792 INFO [IPC Server handler 6 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741847_1023{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:35,813 DEBUG [ProcedureExecutor-11] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns4/test-14769740038903/.tabledesc/.tableinfo.0000000001 2016-10-20 14:33:35,827 INFO [RegionOpenAndInitThread-ns4:test-14769740038903-1] regionserver.HRegion(6162): creating HRegion ns4:test-14769740038903 HTD == 'ns4:test-14769740038903', {NAME => 'f', BLOOMFILTER => 'NONE', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp Table name == ns4:test-14769740038903 2016-10-20 14:33:35,855 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=12 2016-10-20 14:33:35,878 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741848_1024{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:33:35,895 DEBUG [RegionOpenAndInitThread-ns4:test-14769740038903-1] regionserver.HRegion(736): Instantiated ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099. 2016-10-20 14:33:35,896 DEBUG [RegionOpenAndInitThread-ns4:test-14769740038903-1] regionserver.HRegion(1419): Closing ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099.: disabling compactions & flushes 2016-10-20 14:33:35,896 DEBUG [RegionOpenAndInitThread-ns4:test-14769740038903-1] regionserver.HRegion(1446): Updates disabled for region ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099. 2016-10-20 14:33:35,896 INFO [RegionOpenAndInitThread-ns4:test-14769740038903-1] regionserver.HRegion(1552): Closed ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099. 2016-10-20 14:33:36,051 DEBUG [ProcedureExecutor-11] hbase.MetaTableAccessor(1374): Put{"totalColumns":1,"row":"ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099.","families":{"info":[{"qualifier":"regioninfo","vlen":49,"tag":[],"timestamp":9223372036854775807}]}} 2016-10-20 14:33:36,055 INFO [ProcedureExecutor-11] hbase.MetaTableAccessor(1571): Added 1 2016-10-20 14:33:36,158 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=12 2016-10-20 14:33:36,210 INFO [ProcedureExecutor-11] master.AssignmentManager(726): Assigning 1 region(s) to cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:36,211 ERROR [ProcedureExecutor-11] master.TableStateManager(134): Unable to get table ns4:test-14769740038903 state org.apache.hadoop.hbase.TableNotFoundException: ns4:test-14769740038903 at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546) at org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494) 2016-10-20 14:33:36,212 INFO [ProcedureExecutor-11] master.RegionStates(1106): Transition {9d78a3921505ca9bb53c61810d6ae099 state=OFFLINE, ts=1476974016210, server=null} to {9d78a3921505ca9bb53c61810d6ae099 state=PENDING_OPEN, ts=1476974016212, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:33:36,212 INFO [ProcedureExecutor-11] master.RegionStateStore(207): Updating hbase:meta row ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099. with state=PENDING_OPEN, sn=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:36,216 INFO [PriorityRpcServer.handler=3,queue=1,port=37560] regionserver.RSRpcServices(1666): Open ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099. 2016-10-20 14:33:36,227 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(6339): Opening region: {ENCODED => 9d78a3921505ca9bb53c61810d6ae099, NAME => 'ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099.', STARTKEY => '', ENDKEY => ''} 2016-10-20 14:33:36,245 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table test-14769740038903 9d78a3921505ca9bb53c61810d6ae099 2016-10-20 14:33:36,246 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(736): Instantiated ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099. 2016-10-20 14:33:36,252 INFO [StoreOpener-9d78a3921505ca9bb53c61810d6ae099-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:36,254 INFO [StoreOpener-9d78a3921505ca9bb53c61810d6ae099-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:33:36,256 DEBUG [StoreOpener-9d78a3921505ca9bb53c61810d6ae099-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns4/test-14769740038903/9d78a3921505ca9bb53c61810d6ae099/f 2016-10-20 14:33:36,257 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns4/test-14769740038903/9d78a3921505ca9bb53c61810d6ae099 2016-10-20 14:33:36,277 DEBUG [RS_OPEN_REGION-cn012:37560-1] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns4/test-14769740038903/9d78a3921505ca9bb53c61810d6ae099/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:33:36,277 INFO [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(871): Onlined 9d78a3921505ca9bb53c61810d6ae099; next sequenceid=2 2016-10-20 14:33:36,288 INFO [PostOpenDeployTasks:9d78a3921505ca9bb53c61810d6ae099] regionserver.HRegionServer(1957): Post open deploy tasks for ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099. 2016-10-20 14:33:36,289 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.AssignmentManager(2884): Got transition OPENED for {9d78a3921505ca9bb53c61810d6ae099 state=PENDING_OPEN, ts=1476974016212, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:36,289 INFO [B.defaultRpcServer.handler=4,queue=0,port=41289] master.RegionStates(1106): Transition {9d78a3921505ca9bb53c61810d6ae099 state=PENDING_OPEN, ts=1476974016212, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {9d78a3921505ca9bb53c61810d6ae099 state=OPEN, ts=1476974016289, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:33:36,289 INFO [B.defaultRpcServer.handler=4,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099. with state=OPEN, openSeqNum=2, server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:36,292 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.RegionStates(452): Onlined 9d78a3921505ca9bb53c61810d6ae099 on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:36,292 DEBUG [ProcedureExecutor-11] master.AssignmentManager(897): Bulk assigning done for cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:36,292 DEBUG [ProcedureExecutor-11] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"ns4:test-14769740038903","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476974016292}]}} 2016-10-20 14:33:36,292 ERROR [B.defaultRpcServer.handler=4,queue=0,port=41289] master.TableStateManager(134): Unable to get table ns4:test-14769740038903 state org.apache.hadoop.hbase.TableNotFoundException: ns4:test-14769740038903 at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionOpen(AssignmentManager.java:2311) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionTransition(AssignmentManager.java:2891) at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1350) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8623) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2229) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:136) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:111) at java.lang.Thread.run(Thread.java:745) 2016-10-20 14:33:36,293 DEBUG [PostOpenDeployTasks:9d78a3921505ca9bb53c61810d6ae099] regionserver.HRegionServer(1984): Finished post open deploy task for ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099. 2016-10-20 14:33:36,296 DEBUG [RS_OPEN_REGION-cn012:37560-1] handler.OpenRegionHandler(126): Opened ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099. on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:36,298 INFO [ProcedureExecutor-11] hbase.MetaTableAccessor(1700): Updated table ns4:test-14769740038903 state to ENABLED in META 2016-10-20 14:33:36,661 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=12 2016-10-20 14:33:36,812 DEBUG [ProcedureExecutor-11] lock.ZKInterProcessLockBase(328): Released /1/table-lock/ns4:test-14769740038903/write-master:412890000000000 2016-10-20 14:33:36,812 DEBUG [ProcedureExecutor-11] procedure2.ProcedureExecutor(870): Procedure completed in 1.2630sec: CreateTableProcedure (table=ns4:test-14769740038903) id=12 owner=hbase state=FINISHED 2016-10-20 14:33:37,664 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=12 2016-10-20 14:33:37,664 INFO [main] client.HBaseAdmin$TableFuture(3337): Operation: CREATE, Table Name: ns4:test-14769740038903 completed 2016-10-20 14:33:37,665 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2016-10-20 14:33:37,679 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2016-10-20 14:33:37,679 INFO [main] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec6000e 2016-10-20 14:33:37,687 DEBUG [main] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:33:37,693 DEBUG [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Listener(912): RpcServer.listener,port=37560: DISCONNECTING client 172.18.128.12:41932 because read count=-1. Number of active connections: 2 2016-10-20 14:33:37,693 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42562 because read count=-1. Number of active connections: 4 2016-10-20 14:33:37,696 DEBUG [AsyncRpcChannel-pool2-t9] ipc.AsyncRpcChannel$8(566): IPC Client (853659734) to cn012.l42scl.hortonworks.com/172.18.128.12:37560 from hbase: closed 2016-10-20 14:33:37,696 DEBUG [AsyncRpcChannel-pool2-t8] ipc.AsyncRpcChannel$8(566): IPC Client (-171253355) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:33:37,857 INFO [main] hbase.ResourceChecker(148): before: backup.TestIncrementalBackup#TestIncBackupRestore[0] Thread=871, OpenFileDescriptor=1695, MaxFileDescriptor=32768, SystemLoadAverage=76, ProcessCount=337, AvailableMemoryMB=86165 2016-10-20 14:33:37,857 WARN [main] hbase.ResourceChecker(135): Thread=871 is superior to 500 2016-10-20 14:33:37,858 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1695 is superior to 1024 2016-10-20 14:33:37,858 INFO [main] backup.TestIncrementalBackup(77): create full backup image for all tables 2016-10-20 14:33:37,868 INFO [B.defaultRpcServer.handler=1,queue=0,port=41289] master.HMaster(2064): Client=hbase//172.18.128.12 modify ns1:test-1476974003890 2016-10-20 14:33:38,027 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure ModifyTableProcedure (table=ns1:test-1476974003890) id=13 owner=hbase state=RUNNABLE:MODIFY_TABLE_PREPARE added to the store. 2016-10-20 14:33:38,047 DEBUG [ProcedureExecutor-13] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/ns1:test-1476974003890/write-master:412890000000001 2016-10-20 14:33:38,387 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741849_1025{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:38,392 DEBUG [ProcedureExecutor-13] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/.tabledesc/.tableinfo.0000000002 2016-10-20 14:33:38,395 INFO [IPC Server handler 7 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741841_1017 127.0.0.1:38642 2016-10-20 14:33:38,396 DEBUG [ProcedureExecutor-13] util.FSTableDescriptors(664): Deleted table descriptor at hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/.tabledesc/.tableinfo.0000000001 2016-10-20 14:33:38,396 INFO [ProcedureExecutor-13] util.FSTableDescriptors(629): Updated tableinfo=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/.tabledesc/.tableinfo.0000000002 2016-10-20 14:33:38,893 INFO [ProcedureExecutor-13] procedure.MasterDDLOperationHelper(117): Bucketing regions by region server... 2016-10-20 14:33:38,899 INFO [ProcedureExecutor-13] procedure.MasterDDLOperationHelper(146): Reopening 1 regions on 1 region servers. 2016-10-20 14:33:38,901 DEBUG [ProcedureExecutor-13] master.BulkReOpen(57): Creating threads for each region server 2016-10-20 14:33:38,902 INFO [ProcedureExecutor-13] procedure.ModifyTableProcedure(437): Completed modify table operation on table ns1:test-1476974003890 2016-10-20 14:33:38,911 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.AssignmentManager(1352): Starting unassign of ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. (offlining), current state: {4a7f646a7c8fc84833ef454ba337a246 state=OPEN, ts=1476974010204, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:33:38,912 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.RegionStates(1106): Transition {4a7f646a7c8fc84833ef454ba337a246 state=OPEN, ts=1476974010204, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {4a7f646a7c8fc84833ef454ba337a246 state=PENDING_CLOSE, ts=1476974018912, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:33:38,912 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.RegionStateStore(207): Updating hbase:meta row ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. with state=PENDING_CLOSE 2016-10-20 14:33:38,927 INFO [PriorityRpcServer.handler=1,queue=1,port=37560] regionserver.RSRpcServices(1314): Close 4a7f646a7c8fc84833ef454ba337a246, moving to null 2016-10-20 14:33:38,929 DEBUG [RS_CLOSE_REGION-cn012:37560-0] handler.CloseRegionHandler(90): Processing close of ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:38,931 DEBUG [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(1419): Closing ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.: disabling compactions & flushes 2016-10-20 14:33:38,932 DEBUG [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(1446): Updates disabled for region ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:38,932 INFO [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(2345): Flushing 1/1 column families, memstore=16.16 KB 2016-10-20 14:33:38,933 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.AssignmentManager(930): Sent CLOSE to cn012.l42scl.hortonworks.com,37560,1476973974890 for region ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:39,022 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741850_1026{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 8292 2016-10-20 14:33:39,095 DEBUG [ProcedureExecutor-13] lock.ZKInterProcessLockBase(328): Released /1/table-lock/ns1:test-1476974003890/write-master:412890000000001 2016-10-20 14:33:39,095 DEBUG [ProcedureExecutor-13] procedure2.ProcedureExecutor(870): Procedure completed in 1.0320sec: ModifyTableProcedure (table=ns1:test-1476974003890) id=13 owner=hbase state=FINISHED 2016-10-20 14:33:39,302 DEBUG [main] hbase.HBaseTestingUtility(1652): 0/1 regions updated. 2016-10-20 14:33:39,425 INFO [RS_CLOSE_REGION-cn012:37560-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=103, memsize=16.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/.tmp/44766960dbed4b82a69fd46ea16c6a49 2016-10-20 14:33:39,514 DEBUG [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/.tmp/44766960dbed4b82a69fd46ea16c6a49 as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49 2016-10-20 14:33:39,530 INFO [RS_CLOSE_REGION-cn012:37560-0] regionserver.HStore(934): Added hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49, entries=99, sequenceid=103, filesize=8.1 K 2016-10-20 14:33:39,532 INFO [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(2545): Finished memstore flush of ~16.16 KB/16552, currentsize=0 B/0 for region ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. in 600ms, sequenceid=103, compaction requested=false 2016-10-20 14:33:39,539 INFO [StoreCloserThread-ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.-1] regionserver.HStore(839): Closed f 2016-10-20 14:33:39,550 DEBUG [RS_CLOSE_REGION-cn012:37560-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/recovered.edits/106.seqid to file, newSeqId=106, maxSeqId=2 2016-10-20 14:33:39,553 INFO [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(1552): Closed ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:39,556 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.AssignmentManager(2884): Got transition CLOSED for {4a7f646a7c8fc84833ef454ba337a246 state=PENDING_CLOSE, ts=1476974018912, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:39,558 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.RegionStates(1106): Transition {4a7f646a7c8fc84833ef454ba337a246 state=PENDING_CLOSE, ts=1476974018912, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {4a7f646a7c8fc84833ef454ba337a246 state=CLOSED, ts=1476974019558, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:33:39,558 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. with state=CLOSED 2016-10-20 14:33:39,560 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.AssignmentManager(603): Removed region from reopening regions because it was closed 2016-10-20 14:33:39,561 DEBUG [RS_CLOSE_REGION-cn012:37560-0] handler.CloseRegionHandler(122): Closed ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:39,564 DEBUG [AM.-pool3-t1] master.AssignmentManager(1260): Found an existing plan for ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. destination server is cn012.l42scl.hortonworks.com,37560,1476973974890 accepted as a dest server = true 2016-10-20 14:33:39,564 DEBUG [AM.-pool3-t1] master.AssignmentManager(1300): Using pre-existing plan for ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.; plan=hri=ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246., src=, dest=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:39,564 INFO [AM.-pool3-t1] master.AssignmentManager(1080): Assigning ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. to cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:39,564 INFO [AM.-pool3-t1] master.RegionStates(1106): Transition {4a7f646a7c8fc84833ef454ba337a246 state=CLOSED, ts=1476974019558, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {4a7f646a7c8fc84833ef454ba337a246 state=PENDING_OPEN, ts=1476974019564, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:33:39,565 INFO [AM.-pool3-t1] master.RegionStateStore(207): Updating hbase:meta row ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. with state=PENDING_OPEN 2016-10-20 14:33:39,568 INFO [PriorityRpcServer.handler=0,queue=0,port=37560] regionserver.RSRpcServices(1666): Open ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:39,577 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(6339): Opening region: {ENCODED => 4a7f646a7c8fc84833ef454ba337a246, NAME => 'ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.', STARTKEY => '', ENDKEY => ''} 2016-10-20 14:33:39,578 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table test-1476974003890 4a7f646a7c8fc84833ef454ba337a246 2016-10-20 14:33:39,578 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(736): Instantiated ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:39,582 INFO [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:39,583 INFO [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:33:39,601 DEBUG [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] regionserver.HStore(499): loaded hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49, isReference=false, isBulkLoadResult=false, seqid=103, majorCompaction=false 2016-10-20 14:33:39,604 INFO [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:33:39,604 INFO [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:33:39,606 DEBUG [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f3 2016-10-20 14:33:39,609 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246 2016-10-20 14:33:39,610 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@3dddefd8] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:38642 to delete [blk_1073741841_1017] 2016-10-20 14:33:39,612 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.FlushLargeStoresPolicy(72): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table ns1:test-1476974003890, use config (67108864) instead 2016-10-20 14:33:39,617 DEBUG [RS_OPEN_REGION-cn012:37560-2] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/recovered.edits/107.seqid to file, newSeqId=107, maxSeqId=106 2016-10-20 14:33:39,618 INFO [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(871): Onlined 4a7f646a7c8fc84833ef454ba337a246; next sequenceid=107 2016-10-20 14:33:39,621 INFO [PostOpenDeployTasks:4a7f646a7c8fc84833ef454ba337a246] regionserver.HRegionServer(1957): Post open deploy tasks for ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:39,622 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.AssignmentManager(2884): Got transition OPENED for {4a7f646a7c8fc84833ef454ba337a246 state=PENDING_OPEN, ts=1476974019564, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:39,622 INFO [B.defaultRpcServer.handler=3,queue=0,port=41289] master.RegionStates(1106): Transition {4a7f646a7c8fc84833ef454ba337a246 state=PENDING_OPEN, ts=1476974019564, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {4a7f646a7c8fc84833ef454ba337a246 state=OPEN, ts=1476974019622, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:33:39,622 INFO [B.defaultRpcServer.handler=3,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. with state=OPEN, openSeqNum=107, server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:39,625 DEBUG [PostOpenDeployTasks:4a7f646a7c8fc84833ef454ba337a246] regionserver.HRegionServer(1984): Finished post open deploy task for ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:39,628 DEBUG [RS_OPEN_REGION-cn012:37560-2] handler.OpenRegionHandler(126): Opened ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:40,312 DEBUG [main] hbase.HBaseTestingUtility(1652): 1/1 regions updated. 2016-10-20 14:33:41,313 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x2dfb885e connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:33:41,337 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x2dfb885e0x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:33:41,341 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x2dfb885e-0x157e2823ec6000f connected 2016-10-20 14:33:41,341 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e3e1014, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:33:41,342 DEBUG [main] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:33:41,342 DEBUG [main] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:33:41,351 DEBUG [AsyncRpcChannel-pool2-t10] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:33:41,351 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42606; # active connections: 4 2016-10-20 14:33:41,354 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:33:41,355 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42606 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:41,368 DEBUG [AsyncRpcChannel-pool2-t11] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:33:41,370 DEBUG [RpcServer.listener,port=37560] ipc.RpcServer$Listener(880): RpcServer.listener,port=37560: connection from 172.18.128.12:41976; # active connections: 2 2016-10-20 14:33:41,374 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:33:41,374 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 41976 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:41,424 DEBUG [main] backup.BackupInfo(135): CreateBackupContext: 2 ns1:test-1476974003890 2016-10-20 14:33:41,428 INFO [main] impl.FullTableBackupClient(103): Backup backup_1476974021410 started at 1476974021427. 2016-10-20 14:33:41,428 DEBUG [main] impl.BackupSystemTable(123): update backup status in hbase:backup for: backup_1476974021410 set status=RUNNING 2016-10-20 14:33:41,453 DEBUG [main] impl.FullTableBackupClient(107): Backup session backup_1476974021410 has been started. 2016-10-20 14:33:41,453 DEBUG [main] impl.BackupSystemTable(180): read backup start code from hbase:backup 2016-10-20 14:33:41,456 DEBUG [main] impl.BackupSystemTable(205): write backup start code to hbase:backup 0 2016-10-20 14:33:41,465 INFO [main] impl.FullTableBackupClient(479): Execute roll log procedure for full backup ... 2016-10-20 14:33:41,478 DEBUG [AsyncRpcChannel-pool2-t12] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service MasterService, sasl=false 2016-10-20 14:33:41,478 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42612; # active connections: 5 2016-10-20 14:33:41,481 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:33:41,481 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42612 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:41,489 INFO [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(652): Client=hbase//172.18.128.12 procedure request for: rolllog-proc 2016-10-20 14:33:41,500 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] procedure.ProcedureCoordinator(177): Submitting procedure rolllog 2016-10-20 14:33:41,501 INFO [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(196): Starting procedure 'rolllog' 2016-10-20 14:33:41,501 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2016-10-20 14:33:41,503 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(204): Procedure 'rolllog' starting 'acquire' 2016-10-20 14:33:41,504 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(247): Starting procedure 'rolllog', kicking off acquire phase on members. 2016-10-20 14:33:41,505 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/abort/rolllog 2016-10-20 14:33:41,505 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/1/rolllog-proc/acquired/rolllog 2016-10-20 14:33:41,512 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/acquired 2016-10-20 14:33:41,512 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/acquired 2016-10-20 14:33:41,512 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/rolllog-proc/acquired 2016-10-20 14:33:41,512 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/rolllog-proc/acquired' 2016-10-20 14:33:41,512 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:41,512 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/rolllog-proc/acquired 2016-10-20 14:33:41,513 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/rolllog-proc/acquired' 2016-10-20 14:33:41,513 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(188): Found procedure znode: /1/rolllog-proc/acquired/rolllog 2016-10-20 14:33:41,514 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:41,514 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:41,514 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(188): Found procedure znode: /1/rolllog-proc/acquired/rolllog 2016-10-20 14:33:41,514 DEBUG [main-EventThread] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/abort/rolllog 2016-10-20 14:33:41,515 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:41,515 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2016-10-20 14:33:41,515 DEBUG [main-EventThread] zookeeper.ZKUtil(367): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/abort/rolllog 2016-10-20 14:33:41,515 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(214): start proc data length is 35 2016-10-20 14:33:41,515 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(216): Found data for znode:/1/rolllog-proc/acquired/rolllog 2016-10-20 14:33:41,516 INFO [main-EventThread] regionserver.LogRollRegionServerProcedureManager(117): Attempting to run a roll log procedure for backup. 2016-10-20 14:33:41,516 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(214): start proc data length is 35 2016-10-20 14:33:41,516 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(216): Found data for znode:/1/rolllog-proc/acquired/rolllog 2016-10-20 14:33:41,516 INFO [main-EventThread] regionserver.LogRollRegionServerProcedureManager(117): Attempting to run a roll log procedure for backup. 2016-10-20 14:33:41,520 INFO [main-EventThread] regionserver.LogRollBackupSubprocedure(57): Constructing a LogRollBackupSubprocedure. 2016-10-20 14:33:41,520 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:rolllog 2016-10-20 14:33:41,520 INFO [main-EventThread] regionserver.LogRollBackupSubprocedure(57): Constructing a LogRollBackupSubprocedure. 2016-10-20 14:33:41,520 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:rolllog 2016-10-20 14:33:41,521 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(157): Starting subprocedure 'rolllog' with timeout 60000ms 2016-10-20 14:33:41,521 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2016-10-20 14:33:41,521 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(157): Starting subprocedure 'rolllog' with timeout 60000ms 2016-10-20 14:33:41,523 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(165): Subprocedure 'rolllog' starting 'acquire' stage 2016-10-20 14:33:41,523 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2016-10-20 14:33:41,523 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(167): Subprocedure 'rolllog' locally acquired 2016-10-20 14:33:41,525 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(165): Subprocedure 'rolllog' starting 'acquire' stage 2016-10-20 14:33:41,525 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(167): Subprocedure 'rolllog' locally acquired 2016-10-20 14:33:41,525 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.ZKProcedureMemberRpcs(245): Member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' joining acquired barrier for procedure (rolllog) in zk 2016-10-20 14:33:41,525 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.ZKProcedureMemberRpcs(245): Member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' joining acquired barrier for procedure (rolllog) in zk 2016-10-20 14:33:41,528 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:41,529 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:41,528 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.ZKProcedureMemberRpcs(253): Watch for global barrier reached:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:41,529 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:41,529 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.ZKProcedureMemberRpcs(253): Watch for global barrier reached:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:41,529 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:41,529 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/reached/rolllog 2016-10-20 14:33:41,529 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:41,530 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(172): Subprocedure 'rolllog' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2016-10-20 14:33:41,530 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] zookeeper.ZKUtil(367): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/reached/rolllog 2016-10-20 14:33:41,530 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:33:41,530 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(172): Subprocedure 'rolllog' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2016-10-20 14:33:41,531 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:41,531 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:41,532 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:41,533 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:41,533 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:41,534 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:41,534 DEBUG [main-EventThread] procedure.Procedure(298): member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' joining acquired barrier for procedure 'rolllog' on coordinator 2016-10-20 14:33:41,535 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@3986ff6f[Count = 1] remaining members to acquire global barrier 2016-10-20 14:33:41,535 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:41,535 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:41,535 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:41,535 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:41,535 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:41,535 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:33:41,536 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:41,536 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:41,537 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:41,537 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:41,538 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:41,538 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:41,539 DEBUG [main-EventThread] procedure.Procedure(298): member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' joining acquired barrier for procedure 'rolllog' on coordinator 2016-10-20 14:33:41,539 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@3986ff6f[Count = 0] remaining members to acquire global barrier 2016-10-20 14:33:41,539 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(212): Procedure 'rolllog' starting 'in-barrier' execution. 2016-10-20 14:33:41,539 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:41,545 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:41,545 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:41,545 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:41,545 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(134): Recieved reached global barrier:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:41,545 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:41,546 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/reached/rolllog 2016-10-20 14:33:41,546 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:41,546 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:33:41,546 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(186): Subprocedure 'rolllog' received 'reached' from coordinator. 2016-10-20 14:33:41,546 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(134): Recieved reached global barrier:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:41,546 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:41,546 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(186): Subprocedure 'rolllog' received 'reached' from coordinator. 2016-10-20 14:33:41,547 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:41,547 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:41,547 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2016-10-20 14:33:41,547 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:41,548 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] regionserver.LogRollBackupSubprocedurePool(84): Waiting for backup procedure to finish. 2016-10-20 14:33:41,548 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] regionserver.LogRollBackupSubprocedurePool(84): Waiting for backup procedure to finish. 2016-10-20 14:33:41,548 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool19-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(76): ++ DRPC started: cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:41,548 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:41,548 INFO [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool19-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(87): Trying to roll log in backup subprocedure, highest log num: 1476973981466 on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:41,548 DEBUG [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool20-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(76): ++ DRPC started: cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:41,549 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:41,549 INFO [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool20-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(87): Trying to roll log in backup subprocedure, highest log num: 1476973977496 on cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:41,549 DEBUG [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] regionserver.LogRoller(135): WAL roll requested 2016-10-20 14:33:41,549 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:41,550 DEBUG [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] regionserver.LogRoller(135): WAL roll requested 2016-10-20 14:33:41,550 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:41,551 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:41,551 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(238): Ignoring created notification for node:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:41,580 INFO [IPC Server handler 9 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741833_1009{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 2724 2016-10-20 14:33:41,581 INFO [IPC Server handler 4 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741832_1008{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 91 2016-10-20 14:33:41,750 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-jobhistoryserver.properties,hadoop-metrics2.properties 2016-10-20 14:33:41,984 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(884): Rolled WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476973976881 with entries=12, filesize=2.66 KB; new WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974021550 2016-10-20 14:33:41,984 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(884): Rolled WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476973976878 with entries=0, filesize=91 B; new WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974021550 2016-10-20 14:33:41,986 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(951): Archiving hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476973976878 to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476973976878 2016-10-20 14:33:42,029 INFO [IPC Server handler 7 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741838_1014{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 83 2016-10-20 14:33:42,033 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(884): Rolled WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476973981466 with entries=5, filesize=1.55 KB; new WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974021985 2016-10-20 14:33:42,045 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool19-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(94): log roll took 496 2016-10-20 14:33:42,045 INFO [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool19-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(95): After roll log in backup subprocedure, on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,046 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool19-thread-1] impl.BackupSystemTable(222): read region server last roll log result to hbase:backup 2016-10-20 14:33:42,056 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42622; # active connections: 6 2016-10-20 14:33:42,056 DEBUG [AsyncRpcChannel-pool2-t13] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:33:42,062 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase.hfs.0 (auth:SIMPLE) 2016-10-20 14:33:42,062 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42622 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:42,063 INFO [IPC Server handler 6 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741835_1011{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 998 2016-10-20 14:33:42,083 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool19-thread-1] impl.BackupSystemTable(254): write region server last roll log result to hbase:backup 2016-10-20 14:33:42,090 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(188): Subprocedure 'rolllog' locally completed 2016-10-20 14:33:42,091 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.ZKProcedureMemberRpcs(269): Marking procedure 'rolllog' completed for member 'cn012.l42scl.hortonworks.com,37560,1476973974890' in zk 2016-10-20 14:33:42,095 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(193): Subprocedure 'rolllog' has notified controller of completion 2016-10-20 14:33:42,095 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,095 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,096 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,096 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,096 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:42,096 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:33:42,095 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2016-10-20 14:33:42,096 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(218): Subprocedure 'rolllog' completed. 2016-10-20 14:33:42,098 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:42,099 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:42,099 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:42,100 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,101 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,101 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:42,102 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:42,102 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,103 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'rolllog' member 'cn012.l42scl.hortonworks.com,37560,1476973974890': 2016-10-20 14:33:42,103 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' released barrier for procedure'rolllog', counting down latch. Waiting for 1 more 2016-10-20 14:33:42,466 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(884): Rolled WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476973977496 with entries=7, filesize=998 B; new WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974022000 2016-10-20 14:33:42,477 DEBUG [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool20-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(94): log roll took 928 2016-10-20 14:33:42,478 INFO [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool20-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(95): After roll log in backup subprocedure, on cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,478 DEBUG [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool20-thread-1] impl.BackupSystemTable(222): read region server last roll log result to hbase:backup 2016-10-20 14:33:42,490 DEBUG [AsyncRpcChannel-pool2-t14] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:33:42,495 DEBUG [RpcServer.listener,port=37560] ipc.RpcServer$Listener(880): RpcServer.listener,port=37560: connection from 172.18.128.12:41992; # active connections: 3 2016-10-20 14:33:42,496 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:33:42,497 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 41992 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:42,506 DEBUG [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool20-thread-1] impl.BackupSystemTable(254): write region server last roll log result to hbase:backup 2016-10-20 14:33:42,512 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(188): Subprocedure 'rolllog' locally completed 2016-10-20 14:33:42,512 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.ZKProcedureMemberRpcs(269): Marking procedure 'rolllog' completed for member 'cn012.l42scl.hortonworks.com,41289,1476973974467' in zk 2016-10-20 14:33:42,520 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,520 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(193): Subprocedure 'rolllog' has notified controller of completion 2016-10-20 14:33:42,520 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,521 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2016-10-20 14:33:42,521 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,521 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(218): Subprocedure 'rolllog' completed. 2016-10-20 14:33:42,521 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,524 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:42,524 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:33:42,525 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:42,526 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:42,526 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:42,527 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,528 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,529 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:42,529 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:42,530 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,531 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,532 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'rolllog' member 'cn012.l42scl.hortonworks.com,41289,1476973974467': 2016-10-20 14:33:42,532 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' released barrier for procedure'rolllog', counting down latch. Waiting for 0 more 2016-10-20 14:33:42,532 INFO [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(221): Procedure 'rolllog' execution completed 2016-10-20 14:33:42,532 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(230): Running finish phase. 2016-10-20 14:33:42,532 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2016-10-20 14:33:42,533 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:rolllog 2016-10-20 14:33:42,533 INFO [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure rolllogincluding nodes /1/rolllog-proc/acquired /1/rolllog-proc/reached /1/rolllog-proc/abort 2016-10-20 14:33:42,537 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/abort/rolllog 2016-10-20 14:33:42,537 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/abort/rolllog 2016-10-20 14:33:42,537 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/abort/rolllog 2016-10-20 14:33:42,537 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/rolllog-proc/abort/rolllog 2016-10-20 14:33:42,537 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/abort/rolllog 2016-10-20 14:33:42,537 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/rolllog-proc/abort/rolllog 2016-10-20 14:33:42,538 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/abort 2016-10-20 14:33:42,538 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/abort/rolllog 2016-10-20 14:33:42,538 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:42,538 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:33:42,538 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,538 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/rolllog-proc/abort 2016-10-20 14:33:42,539 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/rolllog-proc/abort' 2016-10-20 14:33:42,539 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:42,539 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,540 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/rolllog-proc/abort/rolllog 2016-10-20 14:33:42,540 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:42,540 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:42,541 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:42,542 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,542 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,542 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,542 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,543 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:42,543 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:42,543 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,544 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,545 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/abort 2016-10-20 14:33:42,545 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/rolllog-proc/abort 2016-10-20 14:33:42,545 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/rolllog-proc/abort' 2016-10-20 14:33:42,545 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/rolllog-proc/abort/rolllog 2016-10-20 14:33:42,562 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,562 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog 2016-10-20 14:33:42,562 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/acquired 2016-10-20 14:33:42,562 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/rolllog-proc/acquired 2016-10-20 14:33:42,562 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/rolllog-proc/acquired' 2016-10-20 14:33:42,562 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,563 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog 2016-10-20 14:33:42,563 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/acquired 2016-10-20 14:33:42,563 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/abort 2016-10-20 14:33:42,563 INFO [B.defaultRpcServer.handler=1,queue=0,port=41289] master.LogRollMasterProcedureManager(116): Done waiting - exec procedure for rolllog 2016-10-20 14:33:42,563 INFO [B.defaultRpcServer.handler=1,queue=0,port=41289] master.LogRollMasterProcedureManager(117): Distributed roll log procedure is successful! 2016-10-20 14:33:42,563 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2016-10-20 14:33:42,563 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/rolllog-proc/acquired 2016-10-20 14:33:42,564 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/rolllog-proc/acquired' 2016-10-20 14:33:42,563 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/rolllog-proc/abort 2016-10-20 14:33:42,567 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/rolllog-proc/abort' 2016-10-20 14:33:42,567 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,567 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:42,568 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,568 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:42,568 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/abort/rolllog 2016-10-20 14:33:42,568 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/abort 2016-10-20 14:33:42,568 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/rolllog-proc/abort 2016-10-20 14:33:42,568 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/rolllog-proc/abort' 2016-10-20 14:33:42,569 DEBUG [main] client.HBaseAdmin(2516): Waiting a max of 300000 ms for procedure 'rolllog-proc : rolllog'' to complete. (max 857 ms per retry) 2016-10-20 14:33:42,569 DEBUG [main] client.HBaseAdmin(2525): (#1) Sleeping: 100ms while waiting for procedure completion. 2016-10-20 14:33:42,669 DEBUG [main] client.HBaseAdmin(2531): Getting current status of procedure from master... 2016-10-20 14:33:42,678 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(904): Checking to see if procedure from request:rolllog-proc is done 2016-10-20 14:33:42,681 DEBUG [main] impl.BackupSystemTable(222): read region server last roll log result to hbase:backup 2016-10-20 14:33:42,714 DEBUG [main] impl.BackupSystemTable(599): add WAL files to hbase:backup: backup_1476974021410 hdfs://localhost:37879/backupUT files [hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476973976881,hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476973981466,hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476973977496,hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476973976878] 2016-10-20 14:33:42,714 DEBUG [main] impl.BackupSystemTable(602): add :hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476973976881 2016-10-20 14:33:42,714 DEBUG [main] impl.BackupSystemTable(602): add :hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476973981466 2016-10-20 14:33:42,714 DEBUG [main] impl.BackupSystemTable(602): add :hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476973977496 2016-10-20 14:33:42,715 DEBUG [main] impl.BackupSystemTable(602): add :hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476973976878 2016-10-20 14:33:42,746 INFO [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(1271): Client=hbase//172.18.128.12 snapshot request for:{ ss=snapshot_1476974022731_ns1_test-1476974003890 table=ns1:test-1476974003890 type=FLUSH } 2016-10-20 14:33:42,749 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] snapshot.SnapshotDescriptionUtils(257): Creation time not specified, setting to:1476974022748 (current time:1476974022749). 2016-10-20 14:33:42,750 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] snapshot.SnapshotManager(548): No existing snapshot, attempting snapshot... 2016-10-20 14:33:42,752 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] snapshot.SnapshotManager(592): Table enabled, starting distributed snapshot. 2016-10-20 14:33:42,772 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/ns1:test-1476974003890/write-master:412890000000002 2016-10-20 14:33:42,773 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] snapshot.SnapshotManager(594): Started snapshot: { ss=snapshot_1476974022731_ns1_test-1476974003890 table=ns1:test-1476974003890 type=FLUSH } 2016-10-20 14:33:42,773 INFO [MASTER_TABLE_OPERATIONS-cn012:41289-0] snapshot.TakeSnapshotHandler(166): Running FLUSH table snapshot snapshot_1476974022731_ns1_test-1476974003890 C_M_SNAPSHOT_TABLE on table ns1:test-1476974003890 2016-10-20 14:33:42,779 DEBUG [main] client.HBaseAdmin(2282): Waiting a max of 300000 ms for snapshot '{ ss=snapshot_1476974022731_ns1_test-1476974003890 table=ns1:test-1476974003890 type=FLUSH }'' to complete. (max 857 ms per retry) 2016-10-20 14:33:42,780 DEBUG [main] client.HBaseAdmin(2291): (#1) Sleeping: 100ms while waiting for snapshot completion. 2016-10-20 14:33:42,789 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741855_1031{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:42,796 DEBUG [MASTER_TABLE_OPERATIONS-cn012:41289-0] procedure.ProcedureCoordinator(177): Submitting procedure snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,797 INFO [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(196): Starting procedure 'snapshot_1476974022731_ns1_test-1476974003890' 2016-10-20 14:33:42,797 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 300000 ms 2016-10-20 14:33:42,799 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(204): Procedure 'snapshot_1476974022731_ns1_test-1476974003890' starting 'acquire' 2016-10-20 14:33:42,799 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(247): Starting procedure 'snapshot_1476974022731_ns1_test-1476974003890', kicking off acquire phase on members. 2016-10-20 14:33:42,800 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/online-snapshot/abort/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,800 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,804 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/acquired 2016-10-20 14:33:42,804 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/acquired 2016-10-20 14:33:42,804 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/online-snapshot/acquired 2016-10-20 14:33:42,804 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/online-snapshot/acquired' 2016-10-20 14:33:42,804 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/online-snapshot/acquired 2016-10-20 14:33:42,805 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/online-snapshot/acquired' 2016-10-20 14:33:42,804 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,805 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(188): Found procedure znode: /1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,805 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(188): Found procedure znode: /1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,806 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,806 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2016-10-20 14:33:42,806 DEBUG [main-EventThread] zookeeper.ZKUtil(367): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/online-snapshot/abort/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,806 DEBUG [main-EventThread] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/online-snapshot/abort/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,807 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(214): start proc data length is 86 2016-10-20 14:33:42,807 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(216): Found data for znode:/1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,807 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(214): start proc data length is 86 2016-10-20 14:33:42,807 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(216): Found data for znode:/1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,808 DEBUG [main-EventThread] snapshot.RegionServerSnapshotManager(177): Launching subprocedure for snapshot snapshot_1476974022731_ns1_test-1476974003890 from table ns1:test-1476974003890 type FLUSH 2016-10-20 14:33:42,808 DEBUG [main-EventThread] snapshot.RegionServerSnapshotManager(177): Launching subprocedure for snapshot snapshot_1476974022731_ns1_test-1476974003890 from table ns1:test-1476974003890 type FLUSH 2016-10-20 14:33:42,810 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,811 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,811 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(157): Starting subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' with timeout 300000ms 2016-10-20 14:33:42,811 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 300000 ms 2016-10-20 14:33:42,812 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(157): Starting subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' with timeout 300000ms 2016-10-20 14:33:42,814 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 300000 ms 2016-10-20 14:33:42,814 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(165): Subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' starting 'acquire' stage 2016-10-20 14:33:42,816 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(167): Subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' locally acquired 2016-10-20 14:33:42,816 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.ZKProcedureMemberRpcs(245): Member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' joining acquired barrier for procedure (snapshot_1476974022731_ns1_test-1476974003890) in zk 2016-10-20 14:33:42,816 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(165): Subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' starting 'acquire' stage 2016-10-20 14:33:42,816 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(167): Subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' locally acquired 2016-10-20 14:33:42,816 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.ZKProcedureMemberRpcs(245): Member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' joining acquired barrier for procedure (snapshot_1476974022731_ns1_test-1476974003890) in zk 2016-10-20 14:33:42,820 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,820 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.ZKProcedureMemberRpcs(253): Watch for global barrier reached:/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,820 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,821 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,821 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,821 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:42,821 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/online-snapshot 2016-10-20 14:33:42,829 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.ZKProcedureMemberRpcs(253): Watch for global barrier reached:/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,829 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] zookeeper.ZKUtil(367): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,829 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(172): Subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2016-10-20 14:33:42,829 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:42,829 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,830 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(172): Subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2016-10-20 14:33:42,830 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:42,831 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,832 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,832 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,833 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:42,834 DEBUG [main-EventThread] procedure.Procedure(298): member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' joining acquired barrier for procedure 'snapshot_1476974022731_ns1_test-1476974003890' on coordinator 2016-10-20 14:33:42,834 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@686bab20[Count = 0] remaining members to acquire global barrier 2016-10-20 14:33:42,834 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(212): Procedure 'snapshot_1476974022731_ns1_test-1476974003890' starting 'in-barrier' execution. 2016-10-20 14:33:42,834 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,845 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,845 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,845 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(134): Recieved reached global barrier:/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,846 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,846 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:42,846 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/online-snapshot 2016-10-20 14:33:42,846 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,847 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2016-10-20 14:33:42,845 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,847 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,846 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:42,846 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(186): Subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' received 'reached' from coordinator. 2016-10-20 14:33:42,847 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(134): Recieved reached global barrier:/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,848 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(188): Subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' locally completed 2016-10-20 14:33:42,848 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.ZKProcedureMemberRpcs(269): Marking procedure 'snapshot_1476974022731_ns1_test-1476974003890' completed for member 'cn012.l42scl.hortonworks.com,41289,1476973974467' in zk 2016-10-20 14:33:42,848 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(186): Subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' received 'reached' from coordinator. 2016-10-20 14:33:42,848 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:42,849 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,849 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] snapshot.FlushSnapshotSubprocedure(137): Flush Snapshot Tasks submitted for 1 regions 2016-10-20 14:33:42,850 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool21-thread-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(84): Starting region operation on ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:42,850 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(316): Waiting for local region snapshots to finish. 2016-10-20 14:33:42,850 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool21-thread-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Flush Snapshotting region ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. started... 2016-10-20 14:33:42,852 INFO [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool21-thread-1] regionserver.HRegion(2345): Flushing 2/2 column families, memstore=1.13 KB 2016-10-20 14:33:42,857 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(193): Subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' has notified controller of completion 2016-10-20 14:33:42,857 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:42,857 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2016-10-20 14:33:42,857 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(218): Subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' completed. 2016-10-20 14:33:42,859 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,860 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:42,861 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,862 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:42,862 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(238): Ignoring created notification for node:/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:42,880 INFO [IPC Server handler 9 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741856_1032{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 5284 2016-10-20 14:33:42,880 DEBUG [main] client.HBaseAdmin(2297): Getting current status of snapshot from master... 2016-10-20 14:33:42,884 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.MasterRpcServices(956): Checking to see if snapshot from request:{ ss=snapshot_1476974022731_ns1_test-1476974003890 table=ns1:test-1476974003890 type=FLUSH } is done 2016-10-20 14:33:42,888 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] snapshot.SnapshotManager(377): Snapshoting '{ ss=snapshot_1476974022731_ns1_test-1476974003890 table=ns1:test-1476974003890 type=FLUSH }' is still in progress! 2016-10-20 14:33:42,888 DEBUG [main] client.HBaseAdmin(2291): (#2) Sleeping: 200ms while waiting for snapshot completion. 2016-10-20 14:33:43,089 DEBUG [main] client.HBaseAdmin(2297): Getting current status of snapshot from master... 2016-10-20 14:33:43,091 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.MasterRpcServices(956): Checking to see if snapshot from request:{ ss=snapshot_1476974022731_ns1_test-1476974003890 table=ns1:test-1476974003890 type=FLUSH } is done 2016-10-20 14:33:43,091 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] snapshot.SnapshotManager(377): Snapshoting '{ ss=snapshot_1476974022731_ns1_test-1476974003890 table=ns1:test-1476974003890 type=FLUSH }' is still in progress! 2016-10-20 14:33:43,091 DEBUG [main] client.HBaseAdmin(2291): (#3) Sleeping: 300ms while waiting for snapshot completion. 2016-10-20 14:33:43,282 INFO [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool21-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=115, memsize=1.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/.tmp/0c854275329043169ffbbcd8476ce339 2016-10-20 14:33:43,300 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool21-thread-1] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/.tmp/0c854275329043169ffbbcd8476ce339 as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339 2016-10-20 14:33:43,316 INFO [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool21-thread-1] regionserver.HStore(934): Added hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339, entries=6, sequenceid=115, filesize=5.2 K 2016-10-20 14:33:43,319 INFO [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool21-thread-1] regionserver.HRegion(2545): Finished memstore flush of ~1.13 KB/1152, currentsize=0 B/0 for region ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. in 467ms, sequenceid=115, compaction requested=false 2016-10-20 14:33:43,321 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool21-thread-1] snapshot.SnapshotManifest(203): Storing 'ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.' region-info for snapshot. 2016-10-20 14:33:43,327 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool21-thread-1] snapshot.SnapshotManifest(208): Creating references for hfiles 2016-10-20 14:33:43,332 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool21-thread-1] snapshot.SnapshotManifest(217): Adding snapshot references for [hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49] hfiles 2016-10-20 14:33:43,332 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool21-thread-1] snapshot.SnapshotManifest(226): Adding reference for file (1/1): hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49 2016-10-20 14:33:43,340 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool21-thread-1] snapshot.SnapshotManifest(217): Adding snapshot references for [hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339] hfiles 2016-10-20 14:33:43,340 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool21-thread-1] snapshot.SnapshotManifest(226): Adding reference for file (1/1): hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339 2016-10-20 14:33:43,362 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741857_1033{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:33:43,365 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool21-thread-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(104): ... Flush Snapshotting region ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. completed. 2016-10-20 14:33:43,365 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool21-thread-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(107): Closing region operation on ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:43,366 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(327): Completed 1/1 local region snapshots. 2016-10-20 14:33:43,366 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(329): Completed 1 local region snapshots. 2016-10-20 14:33:43,366 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(361): cancelling 0 tasks for snapshot cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:43,366 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(188): Subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' locally completed 2016-10-20 14:33:43,366 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.ZKProcedureMemberRpcs(269): Marking procedure 'snapshot_1476974022731_ns1_test-1476974003890' completed for member 'cn012.l42scl.hortonworks.com,37560,1476973974890' in zk 2016-10-20 14:33:43,370 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(193): Subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' has notified controller of completion 2016-10-20 14:33:43,370 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:43,370 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2016-10-20 14:33:43,371 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:43,374 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:43,374 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:43,374 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:43,374 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/online-snapshot 2016-10-20 14:33:43,371 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(218): Subprocedure 'snapshot_1476974022731_ns1_test-1476974003890' completed. 2016-10-20 14:33:43,375 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:43,376 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:43,377 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,378 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:43,379 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:43,379 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:43,380 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,381 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:43,382 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:43,383 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'snapshot_1476974022731_ns1_test-1476974003890' member 'cn012.l42scl.hortonworks.com,37560,1476973974890': 2016-10-20 14:33:43,383 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' released barrier for procedure'snapshot_1476974022731_ns1_test-1476974003890', counting down latch. Waiting for 0 more 2016-10-20 14:33:43,383 INFO [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(221): Procedure 'snapshot_1476974022731_ns1_test-1476974003890' execution completed 2016-10-20 14:33:43,383 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(230): Running finish phase. 2016-10-20 14:33:43,384 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2016-10-20 14:33:43,384 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,384 INFO [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure snapshot_1476974022731_ns1_test-1476974003890including nodes /1/online-snapshot/acquired /1/online-snapshot/reached /1/online-snapshot/abort 2016-10-20 14:33:43,392 DEBUG [main] client.HBaseAdmin(2297): Getting current status of snapshot from master... 2016-10-20 14:33:43,394 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.MasterRpcServices(956): Checking to see if snapshot from request:{ ss=snapshot_1476974022731_ns1_test-1476974003890 table=ns1:test-1476974003890 type=FLUSH } is done 2016-10-20 14:33:43,394 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] snapshot.SnapshotManager(377): Snapshoting '{ ss=snapshot_1476974022731_ns1_test-1476974003890 table=ns1:test-1476974003890 type=FLUSH }' is still in progress! 2016-10-20 14:33:43,395 DEBUG [main] client.HBaseAdmin(2291): (#4) Sleeping: 500ms while waiting for snapshot completion. 2016-10-20 14:33:43,395 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/online-snapshot/abort/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,395 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/online-snapshot/abort/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,395 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/online-snapshot/abort/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,395 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/online-snapshot/abort/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,396 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/online-snapshot/abort/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,396 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/online-snapshot/abort/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,396 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/online-snapshot/abort/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,396 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:43,396 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/online-snapshot 2016-10-20 14:33:43,396 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:43,396 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/abort 2016-10-20 14:33:43,397 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/online-snapshot/abort 2016-10-20 14:33:43,397 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/online-snapshot/abort' 2016-10-20 14:33:43,397 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:43,397 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:43,398 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/online-snapshot/abort/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,398 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,399 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:43,399 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,400 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:43,400 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:43,401 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:43,401 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:43,401 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:43,402 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,402 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:43,403 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:43,438 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/acquired 2016-10-20 14:33:43,439 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/online-snapshot/acquired 2016-10-20 14:33:43,439 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/online-snapshot/acquired' 2016-10-20 14:33:43,439 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2016-10-20 14:33:43,439 INFO [MASTER_TABLE_OPERATIONS-cn012:41289-0] snapshot.EnabledTableSnapshotHandler(96): Done waiting - online snapshot for snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,439 DEBUG [main-EventThread] zookeeper.ZKUtil(624): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Unable to get data of znode /1/online-snapshot/abort/snapshot_1476974022731_ns1_test-1476974003890 because node does not exist (not an error) 2016-10-20 14:33:43,442 DEBUG [MASTER_TABLE_OPERATIONS-cn012:41289-0] snapshot.SnapshotManifest(440): Convert to Single Snapshot Manifest 2016-10-20 14:33:43,442 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/abort 2016-10-20 14:33:43,442 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/online-snapshot/abort 2016-10-20 14:33:43,442 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/online-snapshot/abort' 2016-10-20 14:33:43,442 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/abort 2016-10-20 14:33:43,443 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/online-snapshot/abort 2016-10-20 14:33:43,444 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/online-snapshot/abort' 2016-10-20 14:33:43,444 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:43,445 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,445 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:43,445 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/online-snapshot/acquired/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,445 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/acquired 2016-10-20 14:33:43,445 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/online-snapshot/acquired 2016-10-20 14:33:43,445 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/online-snapshot/acquired' 2016-10-20 14:33:43,446 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:43,446 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,447 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:43,447 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/online-snapshot/reached/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,447 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/online-snapshot/abort/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,447 INFO [MASTER_TABLE_OPERATIONS-cn012:41289-0] snapshot.SnapshotManifestV1(119): No regions under directory:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.hbase-snapshot/.tmp/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,478 INFO [IPC Server handler 9 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741858_1034{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:33:43,481 INFO [IPC Server handler 3 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741857_1033 127.0.0.1:38642 2016-10-20 14:33:43,505 DEBUG [MASTER_TABLE_OPERATIONS-cn012:41289-0] snapshot.TakeSnapshotHandler(262): Sentinel is done, just moving the snapshot from hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.hbase-snapshot/.tmp/snapshot_1476974022731_ns1_test-1476974003890 to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.hbase-snapshot/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,507 INFO [MASTER_TABLE_OPERATIONS-cn012:41289-0] snapshot.TakeSnapshotHandler(213): Snapshot snapshot_1476974022731_ns1_test-1476974003890 of table ns1:test-1476974003890 completed 2016-10-20 14:33:43,508 DEBUG [MASTER_TABLE_OPERATIONS-cn012:41289-0] snapshot.TakeSnapshotHandler(226): Launching cleanup of working dir:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.hbase-snapshot/.tmp/snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:43,520 DEBUG [MASTER_TABLE_OPERATIONS-cn012:41289-0] lock.ZKInterProcessLockBase(328): Released /1/table-lock/ns1:test-1476974003890/write-master:412890000000002 2016-10-20 14:33:43,895 DEBUG [main] client.HBaseAdmin(2297): Getting current status of snapshot from master... 2016-10-20 14:33:43,902 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.MasterRpcServices(956): Checking to see if snapshot from request:{ ss=snapshot_1476974022731_ns1_test-1476974003890 table=ns1:test-1476974003890 type=FLUSH } is done 2016-10-20 14:33:43,902 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] snapshot.SnapshotManager(374): Snapshot '{ ss=snapshot_1476974022731_ns1_test-1476974003890 table=ns1:test-1476974003890 type=FLUSH }' has completed, notifying client. 2016-10-20 14:33:43,905 INFO [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(1271): Client=hbase//172.18.128.12 snapshot request for:{ ss=snapshot_1476974023903_ns2_test-14769740038901 table=ns2:test-14769740038901 type=FLUSH } 2016-10-20 14:33:43,905 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] snapshot.SnapshotDescriptionUtils(257): Creation time not specified, setting to:1476974023905 (current time:1476974023905). 2016-10-20 14:33:43,907 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] snapshot.SnapshotManager(548): No existing snapshot, attempting snapshot... 2016-10-20 14:33:43,908 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] snapshot.SnapshotManager(592): Table enabled, starting distributed snapshot. 2016-10-20 14:33:43,930 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/ns2:test-14769740038901/write-master:412890000000001 2016-10-20 14:33:43,931 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] snapshot.SnapshotManager(594): Started snapshot: { ss=snapshot_1476974023903_ns2_test-14769740038901 table=ns2:test-14769740038901 type=FLUSH } 2016-10-20 14:33:43,931 INFO [MASTER_TABLE_OPERATIONS-cn012:41289-0] snapshot.TakeSnapshotHandler(166): Running FLUSH table snapshot snapshot_1476974023903_ns2_test-14769740038901 C_M_SNAPSHOT_TABLE on table ns2:test-14769740038901 2016-10-20 14:33:43,931 DEBUG [main] client.HBaseAdmin(2282): Waiting a max of 300000 ms for snapshot '{ ss=snapshot_1476974023903_ns2_test-14769740038901 table=ns2:test-14769740038901 type=FLUSH }'' to complete. (max 857 ms per retry) 2016-10-20 14:33:43,932 DEBUG [main] client.HBaseAdmin(2291): (#1) Sleeping: 100ms while waiting for snapshot completion. 2016-10-20 14:33:43,962 INFO [IPC Server handler 1 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741859_1035{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 84 2016-10-20 14:33:44,032 DEBUG [main] client.HBaseAdmin(2297): Getting current status of snapshot from master... 2016-10-20 14:33:44,034 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(956): Checking to see if snapshot from request:{ ss=snapshot_1476974023903_ns2_test-14769740038901 table=ns2:test-14769740038901 type=FLUSH } is done 2016-10-20 14:33:44,035 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] snapshot.SnapshotManager(377): Snapshoting '{ ss=snapshot_1476974023903_ns2_test-14769740038901 table=ns2:test-14769740038901 type=FLUSH }' is still in progress! 2016-10-20 14:33:44,035 DEBUG [main] client.HBaseAdmin(2291): (#2) Sleeping: 200ms while waiting for snapshot completion. 2016-10-20 14:33:44,236 DEBUG [main] client.HBaseAdmin(2297): Getting current status of snapshot from master... 2016-10-20 14:33:44,238 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(956): Checking to see if snapshot from request:{ ss=snapshot_1476974023903_ns2_test-14769740038901 table=ns2:test-14769740038901 type=FLUSH } is done 2016-10-20 14:33:44,241 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] snapshot.SnapshotManager(377): Snapshoting '{ ss=snapshot_1476974023903_ns2_test-14769740038901 table=ns2:test-14769740038901 type=FLUSH }' is still in progress! 2016-10-20 14:33:44,243 DEBUG [main] client.HBaseAdmin(2291): (#3) Sleeping: 300ms while waiting for snapshot completion. 2016-10-20 14:33:44,378 DEBUG [MASTER_TABLE_OPERATIONS-cn012:41289-0] procedure.ProcedureCoordinator(177): Submitting procedure snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,378 INFO [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(196): Starting procedure 'snapshot_1476974023903_ns2_test-14769740038901' 2016-10-20 14:33:44,378 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 300000 ms 2016-10-20 14:33:44,378 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(204): Procedure 'snapshot_1476974023903_ns2_test-14769740038901' starting 'acquire' 2016-10-20 14:33:44,379 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(247): Starting procedure 'snapshot_1476974023903_ns2_test-14769740038901', kicking off acquire phase on members. 2016-10-20 14:33:44,380 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/online-snapshot/abort/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,381 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,387 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/acquired 2016-10-20 14:33:44,387 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/acquired 2016-10-20 14:33:44,387 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/online-snapshot/acquired 2016-10-20 14:33:44,388 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/online-snapshot/acquired' 2016-10-20 14:33:44,387 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/online-snapshot/acquired 2016-10-20 14:33:44,388 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/online-snapshot/acquired' 2016-10-20 14:33:44,387 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,389 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(188): Found procedure znode: /1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,389 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(188): Found procedure znode: /1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,389 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,389 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2016-10-20 14:33:44,390 DEBUG [main-EventThread] zookeeper.ZKUtil(367): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/online-snapshot/abort/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,390 DEBUG [main-EventThread] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/online-snapshot/abort/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,391 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(214): start proc data length is 88 2016-10-20 14:33:44,391 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(216): Found data for znode:/1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,391 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(214): start proc data length is 88 2016-10-20 14:33:44,391 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(216): Found data for znode:/1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,391 DEBUG [main-EventThread] snapshot.RegionServerSnapshotManager(177): Launching subprocedure for snapshot snapshot_1476974023903_ns2_test-14769740038901 from table ns2:test-14769740038901 type FLUSH 2016-10-20 14:33:44,391 DEBUG [main-EventThread] snapshot.RegionServerSnapshotManager(177): Launching subprocedure for snapshot snapshot_1476974023903_ns2_test-14769740038901 from table ns2:test-14769740038901 type FLUSH 2016-10-20 14:33:44,392 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,392 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(157): Starting subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' with timeout 300000ms 2016-10-20 14:33:44,393 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 300000 ms 2016-10-20 14:33:44,392 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,405 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(165): Subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' starting 'acquire' stage 2016-10-20 14:33:44,405 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(157): Starting subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' with timeout 300000ms 2016-10-20 14:33:44,406 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 300000 ms 2016-10-20 14:33:44,405 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(167): Subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' locally acquired 2016-10-20 14:33:44,408 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.ZKProcedureMemberRpcs(245): Member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' joining acquired barrier for procedure (snapshot_1476974023903_ns2_test-14769740038901) in zk 2016-10-20 14:33:44,408 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(165): Subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' starting 'acquire' stage 2016-10-20 14:33:44,408 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(167): Subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' locally acquired 2016-10-20 14:33:44,408 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.ZKProcedureMemberRpcs(245): Member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' joining acquired barrier for procedure (snapshot_1476974023903_ns2_test-14769740038901) in zk 2016-10-20 14:33:44,412 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,412 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,412 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,412 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,412 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:44,413 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/online-snapshot 2016-10-20 14:33:44,412 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.ZKProcedureMemberRpcs(253): Watch for global barrier reached:/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,420 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.ZKProcedureMemberRpcs(253): Watch for global barrier reached:/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,420 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:44,420 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] zookeeper.ZKUtil(367): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,421 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(172): Subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2016-10-20 14:33:44,421 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,421 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(172): Subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2016-10-20 14:33:44,421 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:44,422 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,423 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,424 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:44,425 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:44,425 DEBUG [main-EventThread] procedure.Procedure(298): member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' joining acquired barrier for procedure 'snapshot_1476974023903_ns2_test-14769740038901' on coordinator 2016-10-20 14:33:44,426 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@26476904[Count = 0] remaining members to acquire global barrier 2016-10-20 14:33:44,426 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(212): Procedure 'snapshot_1476974023903_ns2_test-14769740038901' starting 'in-barrier' execution. 2016-10-20 14:33:44,427 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,437 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,437 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,437 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,437 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,437 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(134): Recieved reached global barrier:/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,438 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,438 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:44,438 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/online-snapshot 2016-10-20 14:33:44,438 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,438 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2016-10-20 14:33:44,438 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(134): Recieved reached global barrier:/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,438 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(186): Subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' received 'reached' from coordinator. 2016-10-20 14:33:44,439 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(188): Subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' locally completed 2016-10-20 14:33:44,439 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.ZKProcedureMemberRpcs(269): Marking procedure 'snapshot_1476974023903_ns2_test-14769740038901' completed for member 'cn012.l42scl.hortonworks.com,41289,1476973974467' in zk 2016-10-20 14:33:44,439 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:44,439 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(186): Subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' received 'reached' from coordinator. 2016-10-20 14:33:44,440 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:44,440 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] snapshot.FlushSnapshotSubprocedure(137): Flush Snapshot Tasks submitted for 1 regions 2016-10-20 14:33:44,440 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool23-thread-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(84): Starting region operation on ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. 2016-10-20 14:33:44,440 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool23-thread-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(97): Flush Snapshotting region ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. started... 2016-10-20 14:33:44,440 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(316): Waiting for local region snapshots to finish. 2016-10-20 14:33:44,443 INFO [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool23-thread-1] regionserver.HRegion(2345): Flushing 1/1 column families, memstore=16.16 KB 2016-10-20 14:33:44,445 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(193): Subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' has notified controller of completion 2016-10-20 14:33:44,445 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,445 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2016-10-20 14:33:44,446 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1] procedure.Subprocedure(218): Subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' completed. 2016-10-20 14:33:44,448 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,449 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:44,449 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:44,450 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,451 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:44,452 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(238): Ignoring created notification for node:/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,488 INFO [IPC Server handler 6 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741860_1036{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 8292 2016-10-20 14:33:44,544 DEBUG [main] client.HBaseAdmin(2297): Getting current status of snapshot from master... 2016-10-20 14:33:44,547 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(956): Checking to see if snapshot from request:{ ss=snapshot_1476974023903_ns2_test-14769740038901 table=ns2:test-14769740038901 type=FLUSH } is done 2016-10-20 14:33:44,547 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] snapshot.SnapshotManager(377): Snapshoting '{ ss=snapshot_1476974023903_ns2_test-14769740038901 table=ns2:test-14769740038901 type=FLUSH }' is still in progress! 2016-10-20 14:33:44,548 DEBUG [main] client.HBaseAdmin(2291): (#4) Sleeping: 500ms while waiting for snapshot completion. 2016-10-20 14:33:44,890 INFO [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool23-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=103, memsize=16.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/.tmp/a39b76fbbc004d008400803cfd71b6e0 2016-10-20 14:33:44,909 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool23-thread-1] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/.tmp/a39b76fbbc004d008400803cfd71b6e0 as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/f/a39b76fbbc004d008400803cfd71b6e0 2016-10-20 14:33:44,926 INFO [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool23-thread-1] regionserver.HStore(934): Added hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/f/a39b76fbbc004d008400803cfd71b6e0, entries=99, sequenceid=103, filesize=8.1 K 2016-10-20 14:33:44,929 INFO [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool23-thread-1] regionserver.HRegion(2545): Finished memstore flush of ~16.16 KB/16552, currentsize=0 B/0 for region ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. in 486ms, sequenceid=103, compaction requested=false 2016-10-20 14:33:44,929 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool23-thread-1] snapshot.SnapshotManifest(203): Storing 'ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391.' region-info for snapshot. 2016-10-20 14:33:44,929 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool23-thread-1] snapshot.SnapshotManifest(208): Creating references for hfiles 2016-10-20 14:33:44,930 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool23-thread-1] snapshot.SnapshotManifest(217): Adding snapshot references for [hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/f/a39b76fbbc004d008400803cfd71b6e0] hfiles 2016-10-20 14:33:44,930 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool23-thread-1] snapshot.SnapshotManifest(226): Adding reference for file (1/1): hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/f/a39b76fbbc004d008400803cfd71b6e0 2016-10-20 14:33:44,948 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741861_1037{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:33:44,951 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool23-thread-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(104): ... Flush Snapshotting region ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. completed. 2016-10-20 14:33:44,952 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-snapshot-pool23-thread-1] snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(107): Closing region operation on ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. 2016-10-20 14:33:44,952 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(327): Completed 1/1 local region snapshots. 2016-10-20 14:33:44,952 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(329): Completed 1 local region snapshots. 2016-10-20 14:33:44,952 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(361): cancelling 0 tasks for snapshot cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,952 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(188): Subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' locally completed 2016-10-20 14:33:44,952 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.ZKProcedureMemberRpcs(269): Marking procedure 'snapshot_1476974023903_ns2_test-14769740038901' completed for member 'cn012.l42scl.hortonworks.com,37560,1476973974890' in zk 2016-10-20 14:33:44,962 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,962 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(193): Subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' has notified controller of completion 2016-10-20 14:33:44,962 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2016-10-20 14:33:44,962 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1] procedure.Subprocedure(218): Subprocedure 'snapshot_1476974023903_ns2_test-14769740038901' completed. 2016-10-20 14:33:44,962 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,966 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,966 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,966 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:44,967 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/online-snapshot 2016-10-20 14:33:44,967 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:44,968 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:44,969 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,970 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,971 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:44,971 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:44,972 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,973 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,974 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:44,975 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'snapshot_1476974023903_ns2_test-14769740038901' member 'cn012.l42scl.hortonworks.com,37560,1476973974890': 2016-10-20 14:33:44,975 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' released barrier for procedure'snapshot_1476974023903_ns2_test-14769740038901', counting down latch. Waiting for 0 more 2016-10-20 14:33:44,975 INFO [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(221): Procedure 'snapshot_1476974023903_ns2_test-14769740038901' execution completed 2016-10-20 14:33:44,975 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(230): Running finish phase. 2016-10-20 14:33:44,975 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2016-10-20 14:33:44,975 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,975 INFO [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure snapshot_1476974023903_ns2_test-14769740038901including nodes /1/online-snapshot/acquired /1/online-snapshot/reached /1/online-snapshot/abort 2016-10-20 14:33:44,987 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/online-snapshot/abort/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,987 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/online-snapshot/abort/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,987 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/online-snapshot/abort/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,987 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/online-snapshot/abort/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,987 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/online-snapshot/abort/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,987 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/online-snapshot/abort/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,988 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/online-snapshot/abort/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,988 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:44,988 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/online-snapshot 2016-10-20 14:33:44,988 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,988 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/abort 2016-10-20 14:33:44,989 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/online-snapshot/abort 2016-10-20 14:33:44,989 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/online-snapshot/abort' 2016-10-20 14:33:44,989 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:44,989 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:44,989 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/online-snapshot/abort/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,990 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,990 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:44,991 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,992 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,992 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:44,992 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:44,992 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:44,993 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:44,993 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:44,994 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:45,004 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/acquired 2016-10-20 14:33:45,004 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/online-snapshot/acquired 2016-10-20 14:33:45,004 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/online-snapshot/acquired' 2016-10-20 14:33:45,004 INFO [MASTER_TABLE_OPERATIONS-cn012:41289-0] snapshot.EnabledTableSnapshotHandler(96): Done waiting - online snapshot for snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:45,004 DEBUG [MASTER_TABLE_OPERATIONS-cn012:41289-0] snapshot.SnapshotManifest(440): Convert to Single Snapshot Manifest 2016-10-20 14:33:45,004 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2016-10-20 14:33:45,004 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:45,005 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/abort 2016-10-20 14:33:45,007 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/online-snapshot/abort 2016-10-20 14:33:45,007 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/online-snapshot/abort' 2016-10-20 14:33:45,008 DEBUG [main-EventThread] zookeeper.ZKUtil(624): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Unable to get data of znode /1/online-snapshot/abort/snapshot_1476974023903_ns2_test-14769740038901 because node does not exist (not an error) 2016-10-20 14:33:45,008 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/abort 2016-10-20 14:33:45,008 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/online-snapshot/abort 2016-10-20 14:33:45,008 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/online-snapshot/abort' 2016-10-20 14:33:45,008 INFO [MASTER_TABLE_OPERATIONS-cn012:41289-0] snapshot.SnapshotManifestV1(119): No regions under directory:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.hbase-snapshot/.tmp/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:45,009 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:45,009 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:45,009 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:45,009 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/online-snapshot/acquired/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:45,009 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/acquired 2016-10-20 14:33:45,009 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/online-snapshot/acquired 2016-10-20 14:33:45,010 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/online-snapshot/acquired' 2016-10-20 14:33:45,010 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:45,010 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:45,010 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:45,011 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/online-snapshot/reached/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:45,011 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/online-snapshot/abort/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:45,029 INFO [IPC Server handler 9 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741862_1038{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:45,031 INFO [IPC Server handler 6 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741861_1037 127.0.0.1:38642 2016-10-20 14:33:45,047 DEBUG [MASTER_TABLE_OPERATIONS-cn012:41289-0] snapshot.TakeSnapshotHandler(262): Sentinel is done, just moving the snapshot from hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.hbase-snapshot/.tmp/snapshot_1476974023903_ns2_test-14769740038901 to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.hbase-snapshot/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:45,048 DEBUG [main] client.HBaseAdmin(2297): Getting current status of snapshot from master... 2016-10-20 14:33:45,049 INFO [MASTER_TABLE_OPERATIONS-cn012:41289-0] snapshot.TakeSnapshotHandler(213): Snapshot snapshot_1476974023903_ns2_test-14769740038901 of table ns2:test-14769740038901 completed 2016-10-20 14:33:45,049 DEBUG [MASTER_TABLE_OPERATIONS-cn012:41289-0] snapshot.TakeSnapshotHandler(226): Launching cleanup of working dir:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.hbase-snapshot/.tmp/snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:45,050 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(956): Checking to see if snapshot from request:{ ss=snapshot_1476974023903_ns2_test-14769740038901 table=ns2:test-14769740038901 type=FLUSH } is done 2016-10-20 14:33:45,051 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] snapshot.SnapshotManager(374): Snapshot '{ ss=snapshot_1476974023903_ns2_test-14769740038901 table=ns2:test-14769740038901 type=FLUSH }' has completed, notifying client. 2016-10-20 14:33:45,051 DEBUG [main] impl.FullTableBackupClient(509): snapshot copy for backup_1476974021410 2016-10-20 14:33:45,051 INFO [main] impl.FullTableBackupClient(262): Snapshot copy is starting. 2016-10-20 14:33:45,053 DEBUG [MASTER_TABLE_OPERATIONS-cn012:41289-0] lock.ZKInterProcessLockBase(328): Released /1/table-lock/ns2:test-14769740038901/write-master:412890000000001 2016-10-20 14:33:45,054 DEBUG [main] impl.FullTableBackupClient(274): There are 2 snapshots to be copied. 2016-10-20 14:33:45,054 DEBUG [main] impl.FullTableBackupClient(287): Copy snapshot snapshot_1476974023903_ns2_test-14769740038901 to hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901/ 2016-10-20 14:33:45,058 DEBUG [main] mapreduce.MapReduceBackupCopyTask(298): Doing SNAPSHOT_COPY 2016-10-20 14:33:45,085 DEBUG [main] snapshot.ExportSnapshot(929): inputFs=hdfs://localhost:37879 inputRoot=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e 2016-10-20 14:33:45,109 DEBUG [main] snapshot.ExportSnapshot(933): outputFs=hdfs://localhost:37879 outputRoot=hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901 2016-10-20 14:33:45,112 INFO [main] snapshot.ExportSnapshot(977): Copy Snapshot Manifest 2016-10-20 14:33:45,325 INFO [IPC Server handler 9 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741863_1039{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 84 2016-10-20 14:33:45,611 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@3dddefd8] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:38642 to delete [blk_1073741857_1033, blk_1073741861_1037] 2016-10-20 14:33:45,748 INFO [IPC Server handler 0 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741864_1040{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:45,756 WARN [main] mapreduce.TableMapReduceUtil(786): The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it. 2016-10-20 14:33:45,951 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.HConstants, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-5941979194906209094.jar 2016-10-20 14:33:46,843 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-5941666287440446033.jar 2016-10-20 14:33:47,090 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.client.Put, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-4615279633936509146.jar 2016-10-20 14:33:47,106 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.CompatibilityFactory, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-1658681901966108158.jar 2016-10-20 14:33:47,951 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-5256193739906390327.jar 2016-10-20 14:33:47,951 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.zookeeper.ZooKeeper, using jar /home/hbase/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar 2016-10-20 14:33:47,952 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class io.netty.channel.Channel, using jar /home/hbase/.m2/repository/io/netty/netty-all/4.0.30.Final/netty-all-4.0.30.Final.jar 2016-10-20 14:33:47,952 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.protobuf.Message, using jar /home/hbase/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar 2016-10-20 14:33:47,952 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.common.collect.Lists, using jar /home/hbase/.m2/repository/com/google/guava/guava/12.0.1/guava-12.0.1.jar 2016-10-20 14:33:47,953 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.htrace.Trace, using jar /home/hbase/.m2/repository/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar 2016-10-20 14:33:47,953 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.codahale.metrics.MetricRegistry, using jar /home/hbase/.m2/repository/io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.jar 2016-10-20 14:33:47,956 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.io.LongWritable, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-common/2.7.1/hadoop-common-2.7.1.jar 2016-10-20 14:33:47,957 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.io.Text, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-common/2.7.1/hadoop-common-2.7.1.jar 2016-10-20 14:33:47,957 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.mapreduce.lib.input.TextInputFormat, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.1/hadoop-mapreduce-client-core-2.7.1.jar 2016-10-20 14:33:47,958 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.io.LongWritable, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-common/2.7.1/hadoop-common-2.7.1.jar 2016-10-20 14:33:47,958 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.io.Text, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-common/2.7.1/hadoop-common-2.7.1.jar 2016-10-20 14:33:47,959 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.mapreduce.lib.output.TextOutputFormat, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.1/hadoop-mapreduce-client-core-2.7.1.jar 2016-10-20 14:33:47,959 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.1/hadoop-mapreduce-client-core-2.7.1.jar 2016-10-20 14:33:48,032 WARN [main] mapreduce.JobResourceUploader(64): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-10-20 14:33:48,047 WARN [main] mapreduce.JobResourceUploader(171): No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-10-20 14:33:48,459 INFO [main] snapshot.ExportSnapshot(542): Loading Snapshot 'snapshot_1476974023903_ns2_test-14769740038901' hfile list 2016-10-20 14:33:48,473 DEBUG [main] snapshot.ExportSnapshot(629): export split=0 size=8.1 K 2016-10-20 14:33:49,593 INFO [LocalJobRunner Map Task Executor #0] snapshot.ExportSnapshot$ExportMapper(181): Using bufferSize=128 M 2016-10-20 14:33:49,623 INFO [LocalJobRunner Map Task Executor #0] snapshot.ExportSnapshot$ExportMapper(414): copy completed for input=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/f/a39b76fbbc004d008400803cfd71b6e0 output=hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901/archive/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/f/a39b76fbbc004d008400803cfd71b6e0 2016-10-20 14:33:49,623 INFO [LocalJobRunner Map Task Executor #0] snapshot.ExportSnapshot$ExportMapper(415): size=8292 (8.1 K) time=0sec 7.908M/sec 2016-10-20 14:33:49,634 INFO [IPC Server handler 9 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741865_1041{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 8292 2016-10-20 14:33:50,499 INFO [main] snapshot.ExportSnapshot(1007): Finalize the Snapshot Export 2016-10-20 14:33:50,502 INFO [main] snapshot.ExportSnapshot(1018): Verify snapshot integrity 2016-10-20 14:33:50,527 INFO [main] snapshot.ExportSnapshot(1022): Export Completed: snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:50,528 INFO [main] impl.FullTableBackupClient(296): Snapshot copy snapshot_1476974023903_ns2_test-14769740038901 finished. 2016-10-20 14:33:50,528 DEBUG [main] impl.FullTableBackupClient(287): Copy snapshot snapshot_1476974022731_ns1_test-1476974003890 to hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/ 2016-10-20 14:33:50,528 DEBUG [main] mapreduce.MapReduceBackupCopyTask(298): Doing SNAPSHOT_COPY 2016-10-20 14:33:50,560 DEBUG [main] snapshot.ExportSnapshot(929): inputFs=hdfs://localhost:37879 inputRoot=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e 2016-10-20 14:33:50,586 DEBUG [main] snapshot.ExportSnapshot(933): outputFs=hdfs://localhost:37879 outputRoot=hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890 2016-10-20 14:33:50,589 INFO [main] snapshot.ExportSnapshot(977): Copy Snapshot Manifest 2016-10-20 14:33:50,615 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741866_1042{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:50,630 INFO [IPC Server handler 4 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741867_1043{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:50,633 WARN [main] mapreduce.TableMapReduceUtil(786): The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it. 2016-10-20 14:33:50,804 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.HConstants, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-235934766772930045.jar 2016-10-20 14:33:51,677 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-6039815420166069072.jar 2016-10-20 14:33:51,916 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.client.Put, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-2290310639368264991.jar 2016-10-20 14:33:51,930 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.CompatibilityFactory, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-6886059088504113199.jar 2016-10-20 14:33:52,742 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-3231392253058207236.jar 2016-10-20 14:33:52,743 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.zookeeper.ZooKeeper, using jar /home/hbase/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar 2016-10-20 14:33:52,743 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class io.netty.channel.Channel, using jar /home/hbase/.m2/repository/io/netty/netty-all/4.0.30.Final/netty-all-4.0.30.Final.jar 2016-10-20 14:33:52,744 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.protobuf.Message, using jar /home/hbase/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar 2016-10-20 14:33:52,744 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.common.collect.Lists, using jar /home/hbase/.m2/repository/com/google/guava/guava/12.0.1/guava-12.0.1.jar 2016-10-20 14:33:52,744 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.htrace.Trace, using jar /home/hbase/.m2/repository/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar 2016-10-20 14:33:52,745 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.codahale.metrics.MetricRegistry, using jar /home/hbase/.m2/repository/io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.jar 2016-10-20 14:33:52,745 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.io.LongWritable, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-common/2.7.1/hadoop-common-2.7.1.jar 2016-10-20 14:33:52,745 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.io.Text, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-common/2.7.1/hadoop-common-2.7.1.jar 2016-10-20 14:33:52,746 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.mapreduce.lib.input.TextInputFormat, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.1/hadoop-mapreduce-client-core-2.7.1.jar 2016-10-20 14:33:52,746 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.io.LongWritable, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-common/2.7.1/hadoop-common-2.7.1.jar 2016-10-20 14:33:52,747 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.io.Text, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-common/2.7.1/hadoop-common-2.7.1.jar 2016-10-20 14:33:52,747 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.mapreduce.lib.output.TextOutputFormat, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.1/hadoop-mapreduce-client-core-2.7.1.jar 2016-10-20 14:33:52,747 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.1/hadoop-mapreduce-client-core-2.7.1.jar 2016-10-20 14:33:52,798 WARN [main] mapreduce.JobResourceUploader(64): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-10-20 14:33:52,816 WARN [main] mapreduce.JobResourceUploader(171): No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-10-20 14:33:53,203 INFO [main] snapshot.ExportSnapshot(542): Loading Snapshot 'snapshot_1476974022731_ns1_test-1476974003890' hfile list 2016-10-20 14:33:53,207 DEBUG [main] snapshot.ExportSnapshot(629): export split=0 size=13.3 K 2016-10-20 14:33:54,146 INFO [LocalJobRunner Map Task Executor #0] snapshot.ExportSnapshot$ExportMapper(181): Using bufferSize=128 M 2016-10-20 14:33:54,185 INFO [LocalJobRunner Map Task Executor #0] snapshot.ExportSnapshot$ExportMapper(414): copy completed for input=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49 output=hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/archive/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49 2016-10-20 14:33:54,185 INFO [LocalJobRunner Map Task Executor #0] snapshot.ExportSnapshot$ExportMapper(415): size=8292 (8.1 K) time=0sec 7.908M/sec 2016-10-20 14:33:54,198 INFO [IPC Server handler 4 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741868_1044{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:54,371 INFO [LocalJobRunner Map Task Executor #0] snapshot.ExportSnapshot$ExportMapper(414): copy completed for input=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339 output=hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/archive/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339 2016-10-20 14:33:54,373 INFO [LocalJobRunner Map Task Executor #0] snapshot.ExportSnapshot$ExportMapper(415): size=5284 (5.2 K) time=0sec 1.680M/sec 2016-10-20 14:33:54,390 INFO [IPC Server handler 4 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741869_1045{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:55,088 INFO [main] snapshot.ExportSnapshot(1007): Finalize the Snapshot Export 2016-10-20 14:33:55,095 INFO [main] snapshot.ExportSnapshot(1018): Verify snapshot integrity 2016-10-20 14:33:55,127 INFO [main] snapshot.ExportSnapshot(1022): Export Completed: snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:55,127 INFO [main] impl.FullTableBackupClient(296): Snapshot copy snapshot_1476974022731_ns1_test-1476974003890 finished. 2016-10-20 14:33:55,127 DEBUG [main] impl.BackupSystemTable(562): Add incremental backup table set to hbase:backup. ROOT=hdfs://localhost:37879/backupUT tables [ns2:test-14769740038901 ns1:test-1476974003890] 2016-10-20 14:33:55,128 DEBUG [main] impl.BackupSystemTable(565): ns2:test-14769740038901 2016-10-20 14:33:55,128 DEBUG [main] impl.BackupSystemTable(565): ns1:test-1476974003890 2016-10-20 14:33:55,133 DEBUG [main] impl.BackupSystemTable(436): write RS log time stamps to hbase:backup for tables [ns2:test-14769740038901,ns1:test-1476974003890] 2016-10-20 14:33:55,147 DEBUG [main] impl.BackupSystemTable(464): read RS log ts from hbase:backup for root=hdfs://localhost:37879/backupUT 2016-10-20 14:33:55,155 DEBUG [main] impl.BackupSystemTable(205): write backup start code to hbase:backup 1476973977496 2016-10-20 14:33:55,163 DEBUG [main] impl.BackupManifest(477): 1 tables exist in table set. 2016-10-20 14:33:55,163 DEBUG [main] impl.BackupManager(304): Getting the direct ancestors of the current backup backup_1476974021410 2016-10-20 14:33:55,163 DEBUG [main] impl.BackupManager(311): Current backup is a full backup, no direct ancestor for it. 2016-10-20 14:33:55,172 DEBUG [main] impl.BackupManifest(607): hdfs://localhost:37879/backupUT backup_1476974021410 FULL 2016-10-20 14:33:55,193 INFO [IPC Server handler 0 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741870_1046{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 129 2016-10-20 14:33:55,595 INFO [main] impl.BackupManifest(512): Manifest file stored to hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901/.backup.manifest 2016-10-20 14:33:55,595 DEBUG [main] impl.BackupManifest(477): 1 tables exist in table set. 2016-10-20 14:33:55,595 DEBUG [main] impl.BackupManager(304): Getting the direct ancestors of the current backup backup_1476974021410 2016-10-20 14:33:55,595 DEBUG [main] impl.BackupManager(311): Current backup is a full backup, no direct ancestor for it. 2016-10-20 14:33:55,595 DEBUG [main] impl.BackupManifest(607): hdfs://localhost:37879/backupUT backup_1476974021410 FULL 2016-10-20 14:33:55,610 DEBUG [cn012.l42scl.hortonworks.com,37560,1476973974890_ChoreService_1] throttle.PressureAwareCompactionThroughputController(103): compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2016-10-20 14:33:55,612 INFO [IPC Server handler 6 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741871_1047{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:55,613 INFO [main] impl.BackupManifest(512): Manifest file stored to hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/.backup.manifest 2016-10-20 14:33:55,613 DEBUG [main] impl.FullTableBackupClient(408): in-fly convert code here, provided by future jira 2016-10-20 14:33:55,614 DEBUG [main] impl.FullTableBackupClient(416): Backup backup_1476974021410 finished: type=FULL,tablelist=ns2:test-14769740038901;ns1:test-1476974003890,targetRootDir=hdfs://localhost:37879/backupUT,startts=1476974021427,completets=1476974035159,bytescopied=0 2016-10-20 14:33:55,614 DEBUG [main] impl.BackupSystemTable(123): update backup status in hbase:backup for: backup_1476974021410 set status=COMPLETE 2016-10-20 14:33:55,620 DEBUG [main] impl.FullTableBackupClient(127): Trying to delete snapshot for full backup. 2016-10-20 14:33:55,620 DEBUG [main] impl.FullTableBackupClient(132): Trying to delete snapshot: snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:55,631 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(436): Client=hbase//172.18.128.12 delete name: "snapshot_1476974023903_ns2_test-14769740038901" 2016-10-20 14:33:55,637 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] snapshot.SnapshotManager(304): Deleting snapshot: snapshot_1476974023903_ns2_test-14769740038901 2016-10-20 14:33:55,638 INFO [IPC Server handler 7 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741859_1035 127.0.0.1:38642 2016-10-20 14:33:55,639 INFO [IPC Server handler 7 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741862_1038 127.0.0.1:38642 2016-10-20 14:33:55,644 DEBUG [main] impl.FullTableBackupClient(139): Deleting the snapshot snapshot_1476974023903_ns2_test-14769740038901 for backup backup_1476974021410 succeeded. 2016-10-20 14:33:55,645 DEBUG [main] impl.FullTableBackupClient(132): Trying to delete snapshot: snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:55,646 INFO [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(436): Client=hbase//172.18.128.12 delete name: "snapshot_1476974022731_ns1_test-1476974003890" 2016-10-20 14:33:55,652 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] snapshot.SnapshotManager(304): Deleting snapshot: snapshot_1476974022731_ns1_test-1476974003890 2016-10-20 14:33:55,654 INFO [IPC Server handler 4 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741855_1031 127.0.0.1:38642 2016-10-20 14:33:55,654 INFO [IPC Server handler 4 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741858_1034 127.0.0.1:38642 2016-10-20 14:33:55,663 DEBUG [main] impl.FullTableBackupClient(139): Deleting the snapshot snapshot_1476974022731_ns1_test-1476974003890 for backup backup_1476974021410 succeeded. 2016-10-20 14:33:55,668 INFO [main] impl.FullTableBackupClient(431): Backup backup_1476974021410 completed. 2016-10-20 14:33:55,668 DEBUG [main] impl.BackupSystemTable(157): read backup status from hbase:backup for: backup_1476974021410 2016-10-20 14:33:55,676 DEBUG [AsyncRpcChannel-pool2-t15] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:33:55,676 DEBUG [RpcServer.listener,port=37560] ipc.RpcServer$Listener(880): RpcServer.listener,port=37560: connection from 172.18.128.12:42062; # active connections: 4 2016-10-20 14:33:55,679 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:33:55,679 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42062 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:55,698 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467_ChoreService_1] impl.BackupSystemTable(698): Has backup sessions from hbase:backup 2016-10-20 14:33:55,707 DEBUG [AsyncRpcChannel-pool2-t16] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:33:55,707 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42696; # active connections: 7 2016-10-20 14:33:55,709 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:33:55,716 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42696 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:55,725 DEBUG [AsyncRpcChannel-pool2-t17] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:33:55,725 DEBUG [RpcServer.listener,port=37560] ipc.RpcServer$Listener(880): RpcServer.listener,port=37560: connection from 172.18.128.12:42066; # active connections: 5 2016-10-20 14:33:55,727 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:33:55,728 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42066 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:55,905 DEBUG [main] backup.TestIncrementalBackup(100): writing 99 rows to ns1:test-1476974003890 2016-10-20 14:33:55,970 DEBUG [main] backup.TestIncrementalBackup(105): written 99 rows to ns1:test-1476974003890 2016-10-20 14:33:56,013 DEBUG [main] backup.TestIncrementalBackup(117): written 5 rows to ns2:test-14769740038901 2016-10-20 14:33:56,020 DEBUG [AsyncRpcChannel-pool2-t18] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service AdminService, sasl=false 2016-10-20 14:33:56,020 DEBUG [RpcServer.listener,port=37560] ipc.RpcServer$Listener(880): RpcServer.listener,port=37560: connection from 172.18.128.12:42076; # active connections: 6 2016-10-20 14:33:56,023 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:33:56,023 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42076 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:33:56,023 INFO [PriorityRpcServer.handler=0,queue=0,port=37560] regionserver.RSRpcServices(1974): Splitting ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:33:56,024 INFO [PriorityRpcServer.handler=0,queue=0,port=37560] regionserver.HRegion(2345): Flushing 2/2 column families, memstore=18.56 KB 2016-10-20 14:33:56,051 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741872_1048{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 11059 2016-10-20 14:33:56,383 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467_ChoreService_1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns3/test-14769740038902/0bcceb715ec0f6a2bed0aef31b8a540e/f 2016-10-20 14:33:56,385 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467_ChoreService_1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns4/test-14769740038903/9d78a3921505ca9bb53c61810d6ae099/f 2016-10-20 14:33:56,389 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467_ChoreService_1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/backup/7b31f6434ce41b4dbce476e8d5a763e0/meta 2016-10-20 14:33:56,389 DEBUG [region-location-0] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/info 2016-10-20 14:33:56,390 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467_ChoreService_1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/backup/7b31f6434ce41b4dbce476e8d5a763e0/session 2016-10-20 14:33:56,391 DEBUG [region-location-0] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/table 2016-10-20 14:33:56,406 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467_ChoreService_1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/namespace/b386968767678fdb930a9a40fbb05962/info 2016-10-20 14:33:56,407 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467_ChoreService_1] throttle.PressureAwareCompactionThroughputController(103): compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2016-10-20 14:33:56,453 INFO [PriorityRpcServer.handler=0,queue=0,port=37560] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=217, memsize=18.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/.tmp/1beea325f4a642a1a02cce7b95f1e41b 2016-10-20 14:33:56,471 DEBUG [PriorityRpcServer.handler=0,queue=0,port=37560] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/.tmp/1beea325f4a642a1a02cce7b95f1e41b as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f/1beea325f4a642a1a02cce7b95f1e41b 2016-10-20 14:33:56,484 INFO [PriorityRpcServer.handler=0,queue=0,port=37560] regionserver.HStore(934): Added hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f/1beea325f4a642a1a02cce7b95f1e41b, entries=99, sequenceid=217, filesize=10.8 K 2016-10-20 14:33:56,486 INFO [PriorityRpcServer.handler=0,queue=0,port=37560] regionserver.HRegion(2545): Finished memstore flush of ~18.56 KB/19008, currentsize=0 B/0 for region ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. in 462ms, sequenceid=217, compaction requested=false 2016-10-20 14:33:56,491 DEBUG [PriorityRpcServer.handler=0,queue=0,port=37560] regionserver.StoreFile(800): cannot split because midkey is the same as first or last row 2016-10-20 14:33:56,491 DEBUG [PriorityRpcServer.handler=0,queue=0,port=37560] regionserver.StoreFile(800): cannot split because midkey is the same as first or last row 2016-10-20 14:33:56,491 DEBUG [PriorityRpcServer.handler=0,queue=0,port=37560] regionserver.CompactSplitThread(259): Region ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. not splittable because midkey=null 2016-10-20 14:33:56,506 DEBUG [main] client.ConnectionImplementation(604): Table ns1:test-1476974003890 should be available 2016-10-20 14:33:56,506 DEBUG [main] backup.TestIncrementalBackup(133): split finished in =492 2016-10-20 14:33:56,506 DEBUG [main] impl.BackupSystemTable(534): get incr backup table set from hbase:backup Incremental backup for the following table set: [ns2:test-14769740038901, ns1:test-1476974003890] 2016-10-20 14:33:56,516 DEBUG [main] backup.BackupInfo(135): CreateBackupContext: 2 ns2:test-14769740038901 2016-10-20 14:33:56,516 INFO [main] impl.FullTableBackupClient(103): Backup backup_1476974036506 started at 1476974036516. 2016-10-20 14:33:56,516 DEBUG [main] impl.BackupSystemTable(123): update backup status in hbase:backup for: backup_1476974036506 set status=RUNNING 2016-10-20 14:33:56,520 DEBUG [main] impl.FullTableBackupClient(107): Backup session backup_1476974036506 has been started. 2016-10-20 14:33:56,521 DEBUG [main] impl.BackupSystemTable(534): get incr backup table set from hbase:backup 2016-10-20 14:33:56,523 DEBUG [main] impl.IncrementalTableBackupClient(179): For incremental backup, current table set is [ns2:test-14769740038901, ns1:test-1476974003890] 2016-10-20 14:33:56,525 DEBUG [main] impl.BackupSystemTable(180): read backup start code from hbase:backup 2016-10-20 14:33:56,527 DEBUG [main] impl.BackupSystemTable(464): read RS log ts from hbase:backup for root=hdfs://localhost:37879/backupUT 2016-10-20 14:33:56,535 DEBUG [main] impl.IncrementalBackupManager(92): StartCode 1476973977496for backupID backup_1476974036506 2016-10-20 14:33:56,535 INFO [main] impl.IncrementalBackupManager(102): Execute roll log procedure for incremental backup ... 2016-10-20 14:33:56,536 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(652): Client=hbase//172.18.128.12 procedure request for: rolllog-proc 2016-10-20 14:33:56,537 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] procedure.ProcedureCoordinator(155): Procedure rolllog was in running list but was completed. Accepting new attempt. 2016-10-20 14:33:56,537 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] procedure.ProcedureCoordinator(177): Submitting procedure rolllog 2016-10-20 14:33:56,537 INFO [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(196): Starting procedure 'rolllog' 2016-10-20 14:33:56,537 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2016-10-20 14:33:56,540 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(204): Procedure 'rolllog' starting 'acquire' 2016-10-20 14:33:56,540 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(247): Starting procedure 'rolllog', kicking off acquire phase on members. 2016-10-20 14:33:56,540 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/abort/rolllog 2016-10-20 14:33:56,541 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/1/rolllog-proc/acquired/rolllog 2016-10-20 14:33:56,561 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/acquired 2016-10-20 14:33:56,562 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/acquired 2016-10-20 14:33:56,562 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/rolllog-proc/acquired 2016-10-20 14:33:56,562 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/rolllog-proc/acquired' 2016-10-20 14:33:56,562 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/rolllog-proc/acquired 2016-10-20 14:33:56,562 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/rolllog-proc/acquired' 2016-10-20 14:33:56,562 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:56,563 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(188): Found procedure znode: /1/rolllog-proc/acquired/rolllog 2016-10-20 14:33:56,563 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(188): Found procedure znode: /1/rolllog-proc/acquired/rolllog 2016-10-20 14:33:56,563 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:56,563 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:56,563 DEBUG [main-EventThread] zookeeper.ZKUtil(367): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/abort/rolllog 2016-10-20 14:33:56,563 DEBUG [main-EventThread] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/abort/rolllog 2016-10-20 14:33:56,564 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:56,564 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2016-10-20 14:33:56,564 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(214): start proc data length is 35 2016-10-20 14:33:56,564 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(216): Found data for znode:/1/rolllog-proc/acquired/rolllog 2016-10-20 14:33:56,564 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(214): start proc data length is 35 2016-10-20 14:33:56,564 INFO [main-EventThread] regionserver.LogRollRegionServerProcedureManager(117): Attempting to run a roll log procedure for backup. 2016-10-20 14:33:56,564 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(216): Found data for znode:/1/rolllog-proc/acquired/rolllog 2016-10-20 14:33:56,565 INFO [main-EventThread] regionserver.LogRollRegionServerProcedureManager(117): Attempting to run a roll log procedure for backup. 2016-10-20 14:33:56,565 INFO [main-EventThread] regionserver.LogRollBackupSubprocedure(57): Constructing a LogRollBackupSubprocedure. 2016-10-20 14:33:56,565 INFO [main-EventThread] regionserver.LogRollBackupSubprocedure(57): Constructing a LogRollBackupSubprocedure. 2016-10-20 14:33:56,565 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:rolllog 2016-10-20 14:33:56,565 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:rolllog 2016-10-20 14:33:56,565 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(157): Starting subprocedure 'rolllog' with timeout 60000ms 2016-10-20 14:33:56,566 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2016-10-20 14:33:56,565 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(157): Starting subprocedure 'rolllog' with timeout 60000ms 2016-10-20 14:33:56,568 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2016-10-20 14:33:56,567 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(165): Subprocedure 'rolllog' starting 'acquire' stage 2016-10-20 14:33:56,569 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(167): Subprocedure 'rolllog' locally acquired 2016-10-20 14:33:56,569 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.ZKProcedureMemberRpcs(245): Member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' joining acquired barrier for procedure (rolllog) in zk 2016-10-20 14:33:56,569 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(165): Subprocedure 'rolllog' starting 'acquire' stage 2016-10-20 14:33:56,570 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(167): Subprocedure 'rolllog' locally acquired 2016-10-20 14:33:56,570 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.ZKProcedureMemberRpcs(245): Member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' joining acquired barrier for procedure (rolllog) in zk 2016-10-20 14:33:56,578 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:56,578 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:56,578 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:56,578 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.ZKProcedureMemberRpcs(253): Watch for global barrier reached:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:56,578 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:56,578 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:56,579 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:33:56,586 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.ZKProcedureMemberRpcs(253): Watch for global barrier reached:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:56,587 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:56,587 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] zookeeper.ZKUtil(367): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/reached/rolllog 2016-10-20 14:33:56,587 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(172): Subprocedure 'rolllog' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2016-10-20 14:33:56,587 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/reached/rolllog 2016-10-20 14:33:56,587 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:56,587 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(172): Subprocedure 'rolllog' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2016-10-20 14:33:56,588 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:56,589 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:56,589 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:56,590 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:56,591 DEBUG [main-EventThread] procedure.Procedure(298): member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' joining acquired barrier for procedure 'rolllog' on coordinator 2016-10-20 14:33:56,591 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@28349724[Count = 1] remaining members to acquire global barrier 2016-10-20 14:33:56,591 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:56,591 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:56,591 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:56,591 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:56,591 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:56,591 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:33:56,591 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:56,592 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:56,592 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:56,593 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:56,593 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:56,594 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:56,594 DEBUG [main-EventThread] procedure.Procedure(298): member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' joining acquired barrier for procedure 'rolllog' on coordinator 2016-10-20 14:33:56,594 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@28349724[Count = 0] remaining members to acquire global barrier 2016-10-20 14:33:56,594 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(212): Procedure 'rolllog' starting 'in-barrier' execution. 2016-10-20 14:33:56,594 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:56,603 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:56,603 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:56,603 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:56,603 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(134): Recieved reached global barrier:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:56,604 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:56,603 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:56,604 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(134): Recieved reached global barrier:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:56,604 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/reached/rolllog 2016-10-20 14:33:56,604 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:56,604 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:33:56,604 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(186): Subprocedure 'rolllog' received 'reached' from coordinator. 2016-10-20 14:33:56,604 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:56,604 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2016-10-20 14:33:56,605 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool25-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(76): ++ DRPC started: cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:56,604 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(186): Subprocedure 'rolllog' received 'reached' from coordinator. 2016-10-20 14:33:56,605 INFO [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool25-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(87): Trying to roll log in backup subprocedure, highest log num: 1476974021985 on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:56,604 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] regionserver.LogRollBackupSubprocedurePool(84): Waiting for backup procedure to finish. 2016-10-20 14:33:56,604 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:56,605 DEBUG [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool26-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(76): ++ DRPC started: cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:56,605 INFO [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool26-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(87): Trying to roll log in backup subprocedure, highest log num: 1476974022000 on cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:56,605 DEBUG [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] regionserver.LogRoller(135): WAL roll requested 2016-10-20 14:33:56,605 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] regionserver.LogRollBackupSubprocedurePool(84): Waiting for backup procedure to finish. 2016-10-20 14:33:56,605 DEBUG [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] regionserver.LogRoller(135): WAL roll requested 2016-10-20 14:33:56,606 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:56,606 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:56,607 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:56,607 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:56,613 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:56,613 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:56,614 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(238): Ignoring created notification for node:/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:56,665 INFO [IPC Server handler 5 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741852_1028{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 83 2016-10-20 14:33:56,667 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741851_1027{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 14479 2016-10-20 14:33:56,667 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(884): Rolled WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974021550 with entries=0, filesize=91 B; new WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974036606 2016-10-20 14:33:56,668 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(951): Archiving hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974021550 to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974021550 2016-10-20 14:33:56,685 INFO [IPC Server handler 5 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741854_1030{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 91 2016-10-20 14:33:57,069 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(884): Rolled WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974021550 with entries=103, filesize=14.14 KB; new WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974036606 2016-10-20 14:33:57,074 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(951): Archiving hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476973976881 to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476973976881 2016-10-20 14:33:57,079 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(951): Archiving hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974021550 to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974021550 2016-10-20 14:33:57,099 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(884): Rolled WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974022000 with entries=0, filesize=91 B; new WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974036670 2016-10-20 14:33:57,103 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(951): Archiving hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974022000 to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974022000 2016-10-20 14:33:57,109 INFO [IPC Server handler 0 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741853_1029{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 5922 2016-10-20 14:33:57,117 DEBUG [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool26-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(94): log roll took 512 2016-10-20 14:33:57,117 INFO [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool26-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(95): After roll log in backup subprocedure, on cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:57,117 DEBUG [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool26-thread-1] impl.BackupSystemTable(222): read region server last roll log result to hbase:backup 2016-10-20 14:33:57,126 DEBUG [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool26-thread-1] impl.BackupSystemTable(254): write region server last roll log result to hbase:backup 2016-10-20 14:33:57,514 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(884): Rolled WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974021985 with entries=15, filesize=5.78 KB; new WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974037083 2016-10-20 14:33:57,520 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool25-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(94): log roll took 915 2016-10-20 14:33:57,520 INFO [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool25-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(95): After roll log in backup subprocedure, on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:57,520 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool25-thread-1] impl.BackupSystemTable(222): read region server last roll log result to hbase:backup 2016-10-20 14:33:57,523 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(188): Subprocedure 'rolllog' locally completed 2016-10-20 14:33:57,525 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.ZKProcedureMemberRpcs(269): Marking procedure 'rolllog' completed for member 'cn012.l42scl.hortonworks.com,41289,1476973974467' in zk 2016-10-20 14:33:57,529 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool25-thread-1] impl.BackupSystemTable(254): write region server last roll log result to hbase:backup 2016-10-20 14:33:57,532 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(188): Subprocedure 'rolllog' locally completed 2016-10-20 14:33:57,532 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.ZKProcedureMemberRpcs(269): Marking procedure 'rolllog' completed for member 'cn012.l42scl.hortonworks.com,37560,1476973974890' in zk 2016-10-20 14:33:57,537 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:57,537 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(193): Subprocedure 'rolllog' has notified controller of completion 2016-10-20 14:33:57,537 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2016-10-20 14:33:57,537 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:57,537 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(218): Subprocedure 'rolllog' completed. 2016-10-20 14:33:57,540 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:57,540 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:57,540 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:57,540 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:33:57,569 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(193): Subprocedure 'rolllog' has notified controller of completion 2016-10-20 14:33:57,569 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2016-10-20 14:33:57,569 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(218): Subprocedure 'rolllog' completed. 2016-10-20 14:33:57,569 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:57,572 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:57,573 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:57,574 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:57,575 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:57,576 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:57,576 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:57,577 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:57,578 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:57,579 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'rolllog' member 'cn012.l42scl.hortonworks.com,41289,1476973974467': 2016-10-20 14:33:57,579 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' released barrier for procedure'rolllog', counting down latch. Waiting for 1 more 2016-10-20 14:33:57,580 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:57,580 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:57,580 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:57,580 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:57,580 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:57,580 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:33:57,581 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:57,582 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:57,582 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:57,583 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:57,584 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:57,584 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:57,585 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:57,586 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:57,586 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:57,587 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'rolllog' member 'cn012.l42scl.hortonworks.com,37560,1476973974890': 2016-10-20 14:33:57,587 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' released barrier for procedure'rolllog', counting down latch. Waiting for 0 more 2016-10-20 14:33:57,587 INFO [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(221): Procedure 'rolllog' execution completed 2016-10-20 14:33:57,588 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(230): Running finish phase. 2016-10-20 14:33:57,588 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2016-10-20 14:33:57,588 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:rolllog 2016-10-20 14:33:57,588 INFO [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure rolllogincluding nodes /1/rolllog-proc/acquired /1/rolllog-proc/reached /1/rolllog-proc/abort 2016-10-20 14:33:57,603 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/abort/rolllog 2016-10-20 14:33:57,603 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/abort/rolllog 2016-10-20 14:33:57,603 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/rolllog-proc/abort/rolllog 2016-10-20 14:33:57,603 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/abort/rolllog 2016-10-20 14:33:57,604 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/abort/rolllog 2016-10-20 14:33:57,604 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/rolllog-proc/abort/rolllog 2016-10-20 14:33:57,604 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/abort/rolllog 2016-10-20 14:33:57,604 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:33:57,604 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:33:57,605 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:57,605 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/abort 2016-10-20 14:33:57,605 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:33:57,605 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/rolllog-proc/abort 2016-10-20 14:33:57,605 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/rolllog-proc/abort' 2016-10-20 14:33:57,606 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:57,606 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:57,606 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/rolllog-proc/abort/rolllog 2016-10-20 14:33:57,606 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:33:57,607 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:57,608 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:57,608 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:57,608 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:57,609 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:57,609 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:33:57,610 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:33:57,610 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:57,610 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:57,612 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@3dddefd8] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:38642 to delete [blk_1073741858_1034, blk_1073741859_1035, blk_1073741862_1038, blk_1073741855_1031] 2016-10-20 14:33:57,621 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2016-10-20 14:33:57,621 DEBUG [main-EventThread] zookeeper.ZKUtil(624): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Unable to get data of znode /1/rolllog-proc/abort/rolllog because node does not exist (not an error) 2016-10-20 14:33:57,623 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/abort 2016-10-20 14:33:57,623 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/rolllog-proc/abort 2016-10-20 14:33:57,624 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/rolllog-proc/abort' 2016-10-20 14:33:57,621 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/acquired 2016-10-20 14:33:57,624 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/rolllog-proc/acquired 2016-10-20 14:33:57,624 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/rolllog-proc/acquired' 2016-10-20 14:33:57,621 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.LogRollMasterProcedureManager(116): Done waiting - exec procedure for rolllog 2016-10-20 14:33:57,624 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.LogRollMasterProcedureManager(117): Distributed roll log procedure is successful! 2016-10-20 14:33:57,624 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:57,624 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog 2016-10-20 14:33:57,625 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:57,625 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/abort 2016-10-20 14:33:57,625 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/rolllog-proc/abort 2016-10-20 14:33:57,625 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/rolllog-proc/abort' 2016-10-20 14:33:57,625 DEBUG [main] client.HBaseAdmin(2516): Waiting a max of 300000 ms for procedure 'rolllog-proc : rolllog'' to complete. (max 857 ms per retry) 2016-10-20 14:33:57,625 DEBUG [main] client.HBaseAdmin(2525): (#1) Sleeping: 100ms while waiting for procedure completion. 2016-10-20 14:33:57,625 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog 2016-10-20 14:33:57,626 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/acquired 2016-10-20 14:33:57,626 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/rolllog-proc/acquired 2016-10-20 14:33:57,626 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/rolllog-proc/acquired' 2016-10-20 14:33:57,627 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:33:57,627 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:57,627 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:33:57,627 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog 2016-10-20 14:33:57,627 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/abort/rolllog 2016-10-20 14:33:57,726 DEBUG [main] client.HBaseAdmin(2531): Getting current status of procedure from master... 2016-10-20 14:33:57,728 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(904): Checking to see if procedure from request:rolllog-proc is done 2016-10-20 14:33:57,728 DEBUG [main] impl.BackupSystemTable(222): read region server last roll log result to hbase:backup 2016-10-20 14:33:57,747 DEBUG [main] impl.IncrementalBackupManager(214): In getLogFilesForNewBackup() olderTimestamps: {cn012.l42scl.hortonworks.com:41289=1476973977496, cn012.l42scl.hortonworks.com:37560=1476973981466} newestTimestamps: {cn012.l42scl.hortonworks.com:41289=1476974022000, cn012.l42scl.hortonworks.com:37560=1476974021985} 2016-10-20 14:33:57,753 DEBUG [main] impl.IncrementalBackupManager(260): currentLogFile: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974036606 2016-10-20 14:33:57,753 DEBUG [main] impl.IncrementalBackupManager(260): currentLogFile: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974021985 2016-10-20 14:33:57,753 DEBUG [main] impl.IncrementalBackupManager(260): currentLogFile: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974037083 2016-10-20 14:33:57,755 DEBUG [main] impl.IncrementalBackupManager$NewestLogFilter(341): Skip .meta log file: cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.meta.1476973975915.meta 2016-10-20 14:33:57,755 DEBUG [main] impl.IncrementalBackupManager(260): currentLogFile: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974036606 2016-10-20 14:33:57,755 DEBUG [main] impl.IncrementalBackupManager(260): currentLogFile: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974036670 2016-10-20 14:33:57,757 DEBUG [main] impl.BackupSystemTable(619): get WAL files from hbase:backup 2016-10-20 14:33:57,791 INFO [IPC Server handler 3 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741877_1053{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:57,797 DEBUG [main] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/backupUT/backup_1476974036506/ns2/test-14769740038901/.tabledesc/.tableinfo.0000000001 2016-10-20 14:33:57,799 DEBUG [main] util.BackupServerUtil(182): Attempting to copy table info for:ns2:test-14769740038901 target: hdfs://localhost:37879/backupUT/backup_1476974036506/ns2/test-14769740038901 descriptor: TableDescriptor{hTableDescriptor='ns2:test-14769740038901', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}} 2016-10-20 14:33:57,799 DEBUG [main] util.BackupServerUtil(184): Finished copying tableinfo. 2016-10-20 14:33:57,805 DEBUG [main] util.BackupServerUtil(188): Starting to write region info for table ns2:test-14769740038901 2016-10-20 14:33:57,823 INFO [IPC Server handler 0 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741878_1054{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:33:57,827 DEBUG [main] util.BackupServerUtil(197): Finished writing region info for table ns2:test-14769740038901 2016-10-20 14:33:57,847 INFO [IPC Server handler 7 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741879_1055{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:57,851 DEBUG [main] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/backupUT/backup_1476974036506/ns1/test-1476974003890/.tabledesc/.tableinfo.0000000001 2016-10-20 14:33:57,852 DEBUG [main] util.BackupServerUtil(182): Attempting to copy table info for:ns1:test-1476974003890 target: hdfs://localhost:37879/backupUT/backup_1476974036506/ns1/test-1476974003890 descriptor: TableDescriptor{hTableDescriptor='ns1:test-1476974003890', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f3', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}} 2016-10-20 14:33:57,853 DEBUG [main] util.BackupServerUtil(184): Finished copying tableinfo. 2016-10-20 14:33:57,858 DEBUG [main] util.BackupServerUtil(188): Starting to write region info for table ns1:test-1476974003890 2016-10-20 14:33:57,881 INFO [IPC Server handler 9 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741880_1056{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 49 2016-10-20 14:33:58,285 DEBUG [main] util.BackupServerUtil(197): Finished writing region info for table ns1:test-1476974003890 2016-10-20 14:33:58,286 INFO [main] impl.IncrementalTableBackupClient(107): Incremental copy is starting. 2016-10-20 14:33:58,292 DEBUG [main] mapreduce.MapReduceBackupCopyTask(304): Doing COPY_TYPE_DISTCP 2016-10-20 14:33:58,343 DEBUG [main] mapreduce.MapReduceBackupCopyTask(313): DistCp options: [hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974021985, hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974021550, hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974021550, hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974022000, hdfs://localhost:37879/backupUT/backup_1476974036506/WALs] 2016-10-20 14:33:58,465 WARN [main] mapreduce.JobResourceUploader(64): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-10-20 14:33:58,729 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741881_1057{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:33:58,770 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741882_1058{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:33:58,792 INFO [IPC Server handler 1 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741883_1059{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 91 2016-10-20 14:33:59,227 INFO [IPC Server handler 6 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741884_1060{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 91 2016-10-20 14:33:59,658 INFO [main] mapreduce.MapReduceBackupCopyTask$BackupDistCp(228): Progress: 100.0% 2016-10-20 14:33:59,658 DEBUG [main] impl.BackupSystemTable(123): update backup status in hbase:backup for: backup_1476974036506 set status=RUNNING 2016-10-20 14:33:59,663 DEBUG [main] mapreduce.MapReduceBackupCopyTask(137): Backup progress data "100%" has been updated to hbase:backup for backup_1476974036506 2016-10-20 14:33:59,664 DEBUG [main] mapreduce.MapReduceBackupCopyTask$BackupDistCp(231): Backup progress data updated to hbase:backup: "Progress: 100.0%." 2016-10-20 14:34:00,164 INFO [main] mapreduce.MapReduceBackupCopyTask$BackupDistCp(244): Progress: 100.0% subTask: 1.0 mapProgress: 1.0 2016-10-20 14:34:00,165 DEBUG [main] impl.BackupSystemTable(123): update backup status in hbase:backup for: backup_1476974036506 set status=RUNNING 2016-10-20 14:34:00,170 DEBUG [main] mapreduce.MapReduceBackupCopyTask(137): Backup progress data "100%" has been updated to hbase:backup for backup_1476974036506 2016-10-20 14:34:00,170 DEBUG [main] mapreduce.MapReduceBackupCopyTask$BackupDistCp(253): Backup progress data updated to hbase:backup: "Progress: 100.0% - 20583 bytes copied." 2016-10-20 14:34:00,170 DEBUG [main] mapreduce.MapReduceBackupCopyTask$BackupDistCp(267): DistCp job-id: job_local1297389050_0003 completed: true true 2016-10-20 14:34:00,194 DEBUG [main] mapreduce.MapReduceBackupCopyTask$BackupDistCp(270): Counters: 23 File System Counters FILE: Number of bytes read=48123201 FILE: Number of bytes written=47446920 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=104991 HDFS: Number of bytes written=2213267 HDFS: Number of read operations=556 HDFS: Number of large read operations=0 HDFS: Number of write operations=254 Map-Reduce Framework Map input records=4 Map output records=0 Input split bytes=254 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=0 Total committed heap usage (bytes)=2457337856 File Input Format Counters Bytes Read=1520 File Output Format Counters Bytes Written=0 org.apache.hadoop.tools.mapred.CopyMapper$Counter BYTESCOPIED=20583 BYTESEXPECTED=20583 COPY=4 2016-10-20 14:34:00,201 INFO [main] impl.IncrementalTableBackupClient(157): Incremental copy from hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974021985,hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974021550,hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974021550,hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974022000 to hdfs://localhost:37879/backupUT/backup_1476974036506/WALs finished. 2016-10-20 14:34:00,201 DEBUG [main] impl.BackupSystemTable(599): add WAL files to hbase:backup: backup_1476974036506 hdfs://localhost:37879/backupUT files [hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974021985,hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974021550,hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974021550,hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974022000] 2016-10-20 14:34:00,201 DEBUG [main] impl.BackupSystemTable(602): add :hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974021985 2016-10-20 14:34:00,202 DEBUG [main] impl.BackupSystemTable(602): add :hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974021550 2016-10-20 14:34:00,202 DEBUG [main] impl.BackupSystemTable(602): add :hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974021550 2016-10-20 14:34:00,202 DEBUG [main] impl.BackupSystemTable(602): add :hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974022000 2016-10-20 14:34:00,208 DEBUG [main] impl.BackupSystemTable(464): read RS log ts from hbase:backup for root=hdfs://localhost:37879/backupUT 2016-10-20 14:34:00,214 DEBUG [main] impl.BackupSystemTable(436): write RS log time stamps to hbase:backup for tables [ns2:test-14769740038901,ns1:test-1476974003890] 2016-10-20 14:34:00,218 DEBUG [main] impl.BackupSystemTable(464): read RS log ts from hbase:backup for root=hdfs://localhost:37879/backupUT 2016-10-20 14:34:00,224 DEBUG [main] impl.BackupSystemTable(205): write backup start code to hbase:backup 1476974021985 2016-10-20 14:34:00,228 DEBUG [main] impl.BackupManifest(477): 1 tables exist in table set. 2016-10-20 14:34:00,228 DEBUG [main] impl.BackupManager(304): Getting the direct ancestors of the current backup backup_1476974036506 2016-10-20 14:34:00,229 DEBUG [main] impl.BackupSystemTable(271): get backup history from hbase:backup 2016-10-20 14:34:00,229 DEBUG [main] impl.BackupSystemTable(404): get backup contexts from hbase:backup 2016-10-20 14:34:00,236 DEBUG [main] impl.BackupManager(361): Got 1 ancestors for the current backup. 2016-10-20 14:34:00,236 DEBUG [main] impl.BackupManifest(607): hdfs://localhost:37879/backupUT backup_1476974036506 INCREMENTAL 2016-10-20 14:34:00,237 DEBUG [main] impl.BackupManager(304): Getting the direct ancestors of the current backup backup_1476974036506 2016-10-20 14:34:00,237 DEBUG [main] impl.BackupSystemTable(271): get backup history from hbase:backup 2016-10-20 14:34:00,237 DEBUG [main] impl.BackupSystemTable(404): get backup contexts from hbase:backup 2016-10-20 14:34:00,242 DEBUG [main] impl.BackupManager(361): Got 1 ancestors for the current backup. 2016-10-20 14:34:00,253 INFO [IPC Server handler 3 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741885_1061{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:34:00,257 INFO [main] impl.BackupManifest(512): Manifest file stored to hdfs://localhost:37879/backupUT/backup_1476974036506/ns2/test-14769740038901/.backup.manifest 2016-10-20 14:34:00,257 DEBUG [main] impl.BackupManifest(477): 1 tables exist in table set. 2016-10-20 14:34:00,257 DEBUG [main] impl.BackupManager(304): Getting the direct ancestors of the current backup backup_1476974036506 2016-10-20 14:34:00,258 DEBUG [main] impl.BackupSystemTable(271): get backup history from hbase:backup 2016-10-20 14:34:00,258 DEBUG [main] impl.BackupSystemTable(404): get backup contexts from hbase:backup 2016-10-20 14:34:00,264 DEBUG [main] impl.BackupManager(361): Got 1 ancestors for the current backup. 2016-10-20 14:34:00,264 DEBUG [main] impl.BackupManifest(607): hdfs://localhost:37879/backupUT backup_1476974036506 INCREMENTAL 2016-10-20 14:34:00,264 DEBUG [main] impl.BackupManager(304): Getting the direct ancestors of the current backup backup_1476974036506 2016-10-20 14:34:00,264 DEBUG [main] impl.BackupSystemTable(271): get backup history from hbase:backup 2016-10-20 14:34:00,264 DEBUG [main] impl.BackupSystemTable(404): get backup contexts from hbase:backup 2016-10-20 14:34:00,269 DEBUG [main] impl.BackupManager(361): Got 1 ancestors for the current backup. 2016-10-20 14:34:00,278 INFO [IPC Server handler 7 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741886_1062{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:34:00,279 INFO [main] impl.BackupManifest(512): Manifest file stored to hdfs://localhost:37879/backupUT/backup_1476974036506/ns1/test-1476974003890/.backup.manifest 2016-10-20 14:34:00,280 DEBUG [main] impl.BackupManifest(477): 2 tables exist in table set. 2016-10-20 14:34:00,280 DEBUG [main] impl.BackupManager(304): Getting the direct ancestors of the current backup backup_1476974036506 2016-10-20 14:34:00,280 DEBUG [main] impl.BackupSystemTable(271): get backup history from hbase:backup 2016-10-20 14:34:00,280 DEBUG [main] impl.BackupSystemTable(404): get backup contexts from hbase:backup 2016-10-20 14:34:00,285 DEBUG [main] impl.BackupManager(361): Got 1 ancestors for the current backup. 2016-10-20 14:34:00,285 DEBUG [main] impl.BackupManifest(607): hdfs://localhost:37879/backupUT backup_1476974036506 INCREMENTAL 2016-10-20 14:34:00,293 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741887_1063{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:34:00,294 INFO [main] impl.BackupManifest(512): Manifest file stored to hdfs://localhost:37879/backupUT/backup_1476974036506/WALs/.backup.manifest 2016-10-20 14:34:00,295 DEBUG [main] impl.FullTableBackupClient(408): in-fly convert code here, provided by future jira 2016-10-20 14:34:00,295 DEBUG [main] impl.FullTableBackupClient(416): Backup backup_1476974036506 finished: type=INCREMENTAL,tablelist=ns2:test-14769740038901;ns1:test-1476974003890,targetRootDir=hdfs://localhost:37879/backupUT,startts=1476974036516,completets=1476974040228,bytescopied=0 2016-10-20 14:34:00,295 DEBUG [main] impl.BackupSystemTable(123): update backup status in hbase:backup for: backup_1476974036506 set status=COMPLETE 2016-10-20 14:34:00,301 INFO [main] impl.FullTableBackupClient(431): Backup backup_1476974036506 completed. 2016-10-20 14:34:00,301 DEBUG [main] impl.BackupSystemTable(157): read backup status from hbase:backup for: backup_1476974036506 2016-10-20 14:34:00,305 INFO [B.defaultRpcServer.handler=4,queue=0,port=41289] master.HMaster(2064): Client=hbase//172.18.128.12 modify ns1:test-1476974003890 2016-10-20 14:34:00,487 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure ModifyTableProcedure (table=ns1:test-1476974003890) id=14 owner=hbase state=RUNNABLE:MODIFY_TABLE_PREPARE added to the store. 2016-10-20 14:34:00,505 DEBUG [ProcedureExecutor-12] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/ns1:test-1476974003890/write-master:412890000000003 2016-10-20 14:34:00,814 INFO [IPC Server handler 1 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741888_1064{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:34:00,819 DEBUG [ProcedureExecutor-12] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/.tabledesc/.tableinfo.0000000003 2016-10-20 14:34:00,821 INFO [IPC Server handler 3 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741849_1025 127.0.0.1:38642 2016-10-20 14:34:00,822 DEBUG [ProcedureExecutor-12] util.FSTableDescriptors(664): Deleted table descriptor at hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/.tabledesc/.tableinfo.0000000002 2016-10-20 14:34:00,822 INFO [ProcedureExecutor-12] util.FSTableDescriptors(629): Updated tableinfo=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/.tabledesc/.tableinfo.0000000003 2016-10-20 14:34:01,135 DEBUG [ProcedureExecutor-12] procedure.MasterDDLOperationHelper(98): Removing family=f3 from table=ns1:test-1476974003890 2016-10-20 14:34:01,153 DEBUG [ProcedureExecutor-12] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339, to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/archive/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339 2016-10-20 14:34:01,469 INFO [ProcedureExecutor-12] procedure.MasterDDLOperationHelper(117): Bucketing regions by region server... 2016-10-20 14:34:01,473 INFO [ProcedureExecutor-12] procedure.MasterDDLOperationHelper(146): Reopening 1 regions on 1 region servers. 2016-10-20 14:34:01,473 DEBUG [ProcedureExecutor-12] master.BulkReOpen(57): Creating threads for each region server 2016-10-20 14:34:01,474 INFO [ProcedureExecutor-12] procedure.ModifyTableProcedure(437): Completed modify table operation on table ns1:test-1476974003890 2016-10-20 14:34:01,478 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.AssignmentManager(1352): Starting unassign of ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. (offlining), current state: {4a7f646a7c8fc84833ef454ba337a246 state=OPEN, ts=1476974019622, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:01,481 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.RegionStates(1106): Transition {4a7f646a7c8fc84833ef454ba337a246 state=OPEN, ts=1476974019622, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {4a7f646a7c8fc84833ef454ba337a246 state=PENDING_CLOSE, ts=1476974041481, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:01,481 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.RegionStateStore(207): Updating hbase:meta row ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. with state=PENDING_CLOSE 2016-10-20 14:34:01,484 INFO [PriorityRpcServer.handler=3,queue=1,port=37560] regionserver.RSRpcServices(1314): Close 4a7f646a7c8fc84833ef454ba337a246, moving to null 2016-10-20 14:34:01,485 DEBUG [RS_CLOSE_REGION-cn012:37560-1] handler.CloseRegionHandler(90): Processing close of ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:34:01,485 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.AssignmentManager(930): Sent CLOSE to cn012.l42scl.hortonworks.com,37560,1476973974890 for region ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:34:01,485 DEBUG [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(1419): Closing ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.: disabling compactions & flushes 2016-10-20 14:34:01,486 DEBUG [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(1446): Updates disabled for region ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:34:01,494 INFO [StoreCloserThread-ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.-1] regionserver.HStore(839): Closed f 2016-10-20 14:34:01,497 INFO [StoreCloserThread-ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.-1] regionserver.HStore(839): Closed f3 2016-10-20 14:34:01,525 DEBUG [RS_CLOSE_REGION-cn012:37560-1] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/recovered.edits/220.seqid to file, newSeqId=220, maxSeqId=107 2016-10-20 14:34:01,527 INFO [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(1552): Closed ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:34:01,528 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.AssignmentManager(2884): Got transition CLOSED for {4a7f646a7c8fc84833ef454ba337a246 state=PENDING_CLOSE, ts=1476974041481, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:01,530 INFO [B.defaultRpcServer.handler=3,queue=0,port=41289] master.RegionStates(1106): Transition {4a7f646a7c8fc84833ef454ba337a246 state=PENDING_CLOSE, ts=1476974041481, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {4a7f646a7c8fc84833ef454ba337a246 state=CLOSED, ts=1476974041530, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:01,530 INFO [B.defaultRpcServer.handler=3,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. with state=CLOSED 2016-10-20 14:34:01,532 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.AssignmentManager(603): Removed region from reopening regions because it was closed 2016-10-20 14:34:01,533 DEBUG [RS_CLOSE_REGION-cn012:37560-1] handler.CloseRegionHandler(122): Closed ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:34:01,536 DEBUG [AM.-pool3-t2] master.AssignmentManager(1260): Found an existing plan for ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. destination server is cn012.l42scl.hortonworks.com,37560,1476973974890 accepted as a dest server = true 2016-10-20 14:34:01,537 DEBUG [AM.-pool3-t2] master.AssignmentManager(1300): Using pre-existing plan for ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.; plan=hri=ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246., src=, dest=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:01,537 INFO [AM.-pool3-t2] master.AssignmentManager(1080): Assigning ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. to cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:01,537 INFO [AM.-pool3-t2] master.RegionStates(1106): Transition {4a7f646a7c8fc84833ef454ba337a246 state=CLOSED, ts=1476974041530, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {4a7f646a7c8fc84833ef454ba337a246 state=PENDING_OPEN, ts=1476974041537, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:01,537 INFO [AM.-pool3-t2] master.RegionStateStore(207): Updating hbase:meta row ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. with state=PENDING_OPEN 2016-10-20 14:34:01,540 INFO [PriorityRpcServer.handler=0,queue=0,port=37560] regionserver.RSRpcServices(1666): Open ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:34:01,549 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(6339): Opening region: {ENCODED => 4a7f646a7c8fc84833ef454ba337a246, NAME => 'ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.', STARTKEY => '', ENDKEY => ''} 2016-10-20 14:34:01,551 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table test-1476974003890 4a7f646a7c8fc84833ef454ba337a246 2016-10-20 14:34:01,552 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(736): Instantiated ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:34:01,556 INFO [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=3, currentSize=1080968, freeSize=1042881336, maxSize=1043962304, heapSize=1080968, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:01,557 INFO [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:01,574 DEBUG [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] regionserver.HStore(499): loaded hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f/1beea325f4a642a1a02cce7b95f1e41b, isReference=false, isBulkLoadResult=false, seqid=217, majorCompaction=false 2016-10-20 14:34:01,584 DEBUG [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] regionserver.HStore(499): loaded hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49, isReference=false, isBulkLoadResult=false, seqid=103, majorCompaction=false 2016-10-20 14:34:01,587 INFO [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=3, currentSize=1080968, freeSize=1042881336, maxSize=1043962304, heapSize=1080968, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:01,588 INFO [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:01,589 DEBUG [StoreOpener-4a7f646a7c8fc84833ef454ba337a246-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f2 2016-10-20 14:34:01,593 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246 2016-10-20 14:34:01,596 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.FlushLargeStoresPolicy(72): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table ns1:test-1476974003890, use config (67108864) instead 2016-10-20 14:34:01,602 DEBUG [RS_OPEN_REGION-cn012:37560-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/recovered.edits/221.seqid to file, newSeqId=221, maxSeqId=220 2016-10-20 14:34:01,603 INFO [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(871): Onlined 4a7f646a7c8fc84833ef454ba337a246; next sequenceid=221 2016-10-20 14:34:01,604 INFO [PostOpenDeployTasks:4a7f646a7c8fc84833ef454ba337a246] regionserver.HRegionServer(1957): Post open deploy tasks for ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:34:01,606 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.AssignmentManager(2884): Got transition OPENED for {4a7f646a7c8fc84833ef454ba337a246 state=PENDING_OPEN, ts=1476974041537, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:01,606 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.RegionStates(1106): Transition {4a7f646a7c8fc84833ef454ba337a246 state=PENDING_OPEN, ts=1476974041537, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {4a7f646a7c8fc84833ef454ba337a246 state=OPEN, ts=1476974041606, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:01,606 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. with state=OPEN, openSeqNum=221, server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:01,610 DEBUG [PostOpenDeployTasks:4a7f646a7c8fc84833ef454ba337a246] regionserver.HRegionServer(1984): Finished post open deploy task for ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:34:01,613 DEBUG [RS_OPEN_REGION-cn012:37560-0] handler.OpenRegionHandler(126): Opened ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:01,662 DEBUG [ProcedureExecutor-12] lock.ZKInterProcessLockBase(328): Released /1/table-lock/ns1:test-1476974003890/write-master:412890000000003 2016-10-20 14:34:01,662 DEBUG [ProcedureExecutor-12] procedure2.ProcedureExecutor(870): Procedure completed in 1.1690sec: ModifyTableProcedure (table=ns1:test-1476974003890) id=14 owner=hbase state=FINISHED 2016-10-20 14:34:01,746 DEBUG [main] hbase.HBaseTestingUtility(1652): 1/1 regions updated. 2016-10-20 14:34:02,777 DEBUG [main] impl.BackupSystemTable(534): get incr backup table set from hbase:backup Incremental backup for the following table set: [ns2:test-14769740038901, ns1:test-1476974003890] 2016-10-20 14:34:02,786 DEBUG [main] backup.BackupInfo(135): CreateBackupContext: 2 ns2:test-14769740038901 2016-10-20 14:34:02,786 INFO [main] impl.FullTableBackupClient(103): Backup backup_1476974042777 started at 1476974042786. 2016-10-20 14:34:02,786 DEBUG [main] impl.BackupSystemTable(123): update backup status in hbase:backup for: backup_1476974042777 set status=RUNNING 2016-10-20 14:34:02,790 DEBUG [main] impl.FullTableBackupClient(107): Backup session backup_1476974042777 has been started. 2016-10-20 14:34:02,791 DEBUG [main] impl.BackupSystemTable(534): get incr backup table set from hbase:backup 2016-10-20 14:34:02,792 DEBUG [main] impl.IncrementalTableBackupClient(179): For incremental backup, current table set is [ns2:test-14769740038901, ns1:test-1476974003890] 2016-10-20 14:34:02,793 DEBUG [main] impl.BackupSystemTable(180): read backup start code from hbase:backup 2016-10-20 14:34:02,794 DEBUG [main] impl.BackupSystemTable(464): read RS log ts from hbase:backup for root=hdfs://localhost:37879/backupUT 2016-10-20 14:34:02,800 DEBUG [main] impl.IncrementalBackupManager(92): StartCode 1476974021985for backupID backup_1476974042777 2016-10-20 14:34:02,800 INFO [main] impl.IncrementalBackupManager(102): Execute roll log procedure for incremental backup ... 2016-10-20 14:34:02,802 INFO [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(652): Client=hbase//172.18.128.12 procedure request for: rolllog-proc 2016-10-20 14:34:02,802 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] procedure.ProcedureCoordinator(155): Procedure rolllog was in running list but was completed. Accepting new attempt. 2016-10-20 14:34:02,803 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] procedure.ProcedureCoordinator(177): Submitting procedure rolllog 2016-10-20 14:34:02,803 INFO [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(196): Starting procedure 'rolllog' 2016-10-20 14:34:02,803 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2016-10-20 14:34:02,806 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(204): Procedure 'rolllog' starting 'acquire' 2016-10-20 14:34:02,806 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(247): Starting procedure 'rolllog', kicking off acquire phase on members. 2016-10-20 14:34:02,807 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/abort/rolllog 2016-10-20 14:34:02,807 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/1/rolllog-proc/acquired/rolllog 2016-10-20 14:34:02,811 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:02,811 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/acquired 2016-10-20 14:34:02,811 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/acquired 2016-10-20 14:34:02,812 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/rolllog-proc/acquired 2016-10-20 14:34:02,812 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/rolllog-proc/acquired' 2016-10-20 14:34:02,812 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:02,812 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:02,812 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/rolllog-proc/acquired 2016-10-20 14:34:02,813 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/rolllog-proc/acquired' 2016-10-20 14:34:02,813 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(188): Found procedure znode: /1/rolllog-proc/acquired/rolllog 2016-10-20 14:34:02,813 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:02,813 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2016-10-20 14:34:02,813 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(188): Found procedure znode: /1/rolllog-proc/acquired/rolllog 2016-10-20 14:34:02,814 DEBUG [main-EventThread] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/abort/rolllog 2016-10-20 14:34:02,814 DEBUG [main-EventThread] zookeeper.ZKUtil(367): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/abort/rolllog 2016-10-20 14:34:02,814 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(214): start proc data length is 35 2016-10-20 14:34:02,815 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(216): Found data for znode:/1/rolllog-proc/acquired/rolllog 2016-10-20 14:34:02,815 INFO [main-EventThread] regionserver.LogRollRegionServerProcedureManager(117): Attempting to run a roll log procedure for backup. 2016-10-20 14:34:02,815 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(214): start proc data length is 35 2016-10-20 14:34:02,815 INFO [main-EventThread] regionserver.LogRollBackupSubprocedure(57): Constructing a LogRollBackupSubprocedure. 2016-10-20 14:34:02,816 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(216): Found data for znode:/1/rolllog-proc/acquired/rolllog 2016-10-20 14:34:02,816 WARN [main-EventThread] procedure.ProcedureMember(142): A completed old subproc rolllog is still present, removing 2016-10-20 14:34:02,816 INFO [main-EventThread] regionserver.LogRollRegionServerProcedureManager(117): Attempting to run a roll log procedure for backup. 2016-10-20 14:34:02,816 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:rolllog 2016-10-20 14:34:02,816 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(157): Starting subprocedure 'rolllog' with timeout 60000ms 2016-10-20 14:34:02,817 INFO [main-EventThread] regionserver.LogRollBackupSubprocedure(57): Constructing a LogRollBackupSubprocedure. 2016-10-20 14:34:02,817 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2016-10-20 14:34:02,817 WARN [main-EventThread] procedure.ProcedureMember(142): A completed old subproc rolllog is still present, removing 2016-10-20 14:34:02,819 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(165): Subprocedure 'rolllog' starting 'acquire' stage 2016-10-20 14:34:02,819 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(167): Subprocedure 'rolllog' locally acquired 2016-10-20 14:34:02,819 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.ZKProcedureMemberRpcs(245): Member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' joining acquired barrier for procedure (rolllog) in zk 2016-10-20 14:34:02,819 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:rolllog 2016-10-20 14:34:02,820 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(157): Starting subprocedure 'rolllog' with timeout 60000ms 2016-10-20 14:34:02,820 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2016-10-20 14:34:02,822 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(165): Subprocedure 'rolllog' starting 'acquire' stage 2016-10-20 14:34:02,822 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(167): Subprocedure 'rolllog' locally acquired 2016-10-20 14:34:02,822 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.ZKProcedureMemberRpcs(245): Member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' joining acquired barrier for procedure (rolllog) in zk 2016-10-20 14:34:02,828 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:02,828 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.ZKProcedureMemberRpcs(253): Watch for global barrier reached:/1/rolllog-proc/reached/rolllog 2016-10-20 14:34:02,828 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:02,828 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:02,829 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:02,829 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:34:02,829 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:34:02,836 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.ZKProcedureMemberRpcs(253): Watch for global barrier reached:/1/rolllog-proc/reached/rolllog 2016-10-20 14:34:02,836 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/reached/rolllog 2016-10-20 14:34:02,837 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(172): Subprocedure 'rolllog' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2016-10-20 14:34:02,836 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:34:02,837 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] zookeeper.ZKUtil(367): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/reached/rolllog 2016-10-20 14:34:02,837 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(172): Subprocedure 'rolllog' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2016-10-20 14:34:02,838 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:34:02,838 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:34:02,839 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:02,840 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:02,840 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:34:02,841 DEBUG [main-EventThread] procedure.Procedure(298): member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' joining acquired barrier for procedure 'rolllog' on coordinator 2016-10-20 14:34:02,841 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@59223545[Count = 1] remaining members to acquire global barrier 2016-10-20 14:34:02,841 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:02,841 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:02,841 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:02,841 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:02,841 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:34:02,842 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:34:02,842 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:34:02,843 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:34:02,843 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:34:02,844 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:02,844 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:02,845 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:34:02,845 DEBUG [main-EventThread] procedure.Procedure(298): member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' joining acquired barrier for procedure 'rolllog' on coordinator 2016-10-20 14:34:02,845 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@59223545[Count = 0] remaining members to acquire global barrier 2016-10-20 14:34:02,845 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(212): Procedure 'rolllog' starting 'in-barrier' execution. 2016-10-20 14:34:02,845 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/1/rolllog-proc/reached/rolllog 2016-10-20 14:34:02,853 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog 2016-10-20 14:34:02,853 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/reached/rolllog 2016-10-20 14:34:02,853 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(134): Recieved reached global barrier:/1/rolllog-proc/reached/rolllog 2016-10-20 14:34:02,853 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog 2016-10-20 14:34:02,853 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(186): Subprocedure 'rolllog' received 'reached' from coordinator. 2016-10-20 14:34:02,853 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/reached/rolllog 2016-10-20 14:34:02,854 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:34:02,854 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:34:02,854 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/reached/rolllog 2016-10-20 14:34:02,854 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(134): Recieved reached global barrier:/1/rolllog-proc/reached/rolllog 2016-10-20 14:34:02,854 DEBUG [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool27-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(76): ++ DRPC started: cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:02,855 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:34:02,854 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] regionserver.LogRollBackupSubprocedurePool(84): Waiting for backup procedure to finish. 2016-10-20 14:34:02,854 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:02,855 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:34:02,855 INFO [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool27-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(87): Trying to roll log in backup subprocedure, highest log num: 1476974036670 on cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:02,854 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(186): Subprocedure 'rolllog' received 'reached' from coordinator. 2016-10-20 14:34:02,856 DEBUG [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] regionserver.LogRoller(135): WAL roll requested 2016-10-20 14:34:02,856 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:34:02,856 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:02,856 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2016-10-20 14:34:02,856 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool28-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(76): ++ DRPC started: cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:02,856 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] regionserver.LogRollBackupSubprocedurePool(84): Waiting for backup procedure to finish. 2016-10-20 14:34:02,857 INFO [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool28-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(87): Trying to roll log in backup subprocedure, highest log num: 1476974037083 on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:02,857 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:02,857 DEBUG [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] regionserver.LogRoller(135): WAL roll requested 2016-10-20 14:34:02,858 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:02,858 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:34:02,859 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:34:02,859 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(238): Ignoring created notification for node:/1/rolllog-proc/reached/rolllog 2016-10-20 14:34:02,875 INFO [IPC Server handler 3 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741874_1050{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 1734 2016-10-20 14:34:02,877 INFO [IPC Server handler 6 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741873_1049{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 91 2016-10-20 14:34:03,278 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(884): Rolled WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974036606 with entries=9, filesize=1.69 KB; new WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974042857 2016-10-20 14:34:03,279 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(884): Rolled WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974036606 with entries=0, filesize=91 B; new WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974042856 2016-10-20 14:34:03,280 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(951): Archiving hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974036606 to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974036606 2016-10-20 14:34:03,300 INFO [IPC Server handler 0 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741876_1052{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 5239 2016-10-20 14:34:03,308 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741875_1051{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 91 2016-10-20 14:34:03,612 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@3dddefd8] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:38642 to delete [blk_1073741849_1025] 2016-10-20 14:34:03,706 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(884): Rolled WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974037083 with entries=9, filesize=5.12 KB; new WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974043279 2016-10-20 14:34:03,711 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(884): Rolled WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974036670 with entries=0, filesize=91 B; new WAL /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974043284 2016-10-20 14:34:03,712 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] wal.FSHLog(951): Archiving hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974036670 to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974036670 2016-10-20 14:34:03,720 DEBUG [cn012.l42scl.hortonworks.com,34094,1476973981831_ChoreService_1] throttle.PressureAwareCompactionThroughputController(103): compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2016-10-20 14:34:03,723 DEBUG [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool27-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(94): log roll took 867 2016-10-20 14:34:03,723 INFO [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool27-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(95): After roll log in backup subprocedure, on cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:03,723 DEBUG [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool27-thread-1] impl.BackupSystemTable(222): read region server last roll log result to hbase:backup 2016-10-20 14:34:03,723 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool28-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(94): log roll took 866 2016-10-20 14:34:03,724 INFO [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool28-thread-1] regionserver.LogRollBackupSubprocedure$RSRollLogTask(95): After roll log in backup subprocedure, on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:03,724 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool28-thread-1] impl.BackupSystemTable(222): read region server last roll log result to hbase:backup 2016-10-20 14:34:03,728 DEBUG [rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool28-thread-1] impl.BackupSystemTable(254): write region server last roll log result to hbase:backup 2016-10-20 14:34:03,734 DEBUG [rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool27-thread-1] impl.BackupSystemTable(254): write region server last roll log result to hbase:backup 2016-10-20 14:34:03,735 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(188): Subprocedure 'rolllog' locally completed 2016-10-20 14:34:03,736 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.ZKProcedureMemberRpcs(269): Marking procedure 'rolllog' completed for member 'cn012.l42scl.hortonworks.com,37560,1476973974890' in zk 2016-10-20 14:34:03,737 DEBUG [cn012.l42scl.hortonworks.com,34206,1476973982007_ChoreService_1] throttle.PressureAwareCompactionThroughputController(103): compactionPressure is 0.0, tune compaction throughput to 10.00 MB/sec 2016-10-20 14:34:03,739 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(188): Subprocedure 'rolllog' locally completed 2016-10-20 14:34:03,739 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.ZKProcedureMemberRpcs(269): Marking procedure 'rolllog' completed for member 'cn012.l42scl.hortonworks.com,41289,1476973974467' in zk 2016-10-20 14:34:04,203 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:04,203 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(193): Subprocedure 'rolllog' has notified controller of completion 2016-10-20 14:34:04,203 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:04,204 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:04,204 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:04,204 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:34:04,204 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:34:04,204 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2016-10-20 14:34:04,204 DEBUG [member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1] procedure.Subprocedure(218): Subprocedure 'rolllog' completed. 2016-10-20 14:34:04,211 DEBUG [cn012.l42scl.hortonworks.com,34094,1476973981831_ChoreService_1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/backup/4a6cefc1b47013b7aa8a0dd5a0a290e5/meta 2016-10-20 14:34:04,211 DEBUG [region-location-0] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/info 2016-10-20 14:34:04,213 DEBUG [cn012.l42scl.hortonworks.com,34094,1476973981831_ChoreService_1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/backup/4a6cefc1b47013b7aa8a0dd5a0a290e5/session 2016-10-20 14:34:04,213 DEBUG [region-location-0] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/table 2016-10-20 14:34:04,215 DEBUG [cn012.l42scl.hortonworks.com,34094,1476973981831_ChoreService_1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/namespace/37fa3cc8df36677363de1aacf6de184f/info 2016-10-20 14:34:04,284 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(193): Subprocedure 'rolllog' has notified controller of completion 2016-10-20 14:34:04,285 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2016-10-20 14:34:04,285 DEBUG [member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1] procedure.Subprocedure(218): Subprocedure 'rolllog' completed. 2016-10-20 14:34:04,284 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:34:04,288 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:34:04,289 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:34:04,290 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:04,290 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:04,291 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:34:04,292 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:34:04,293 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:04,293 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:04,294 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'rolllog' member 'cn012.l42scl.hortonworks.com,37560,1476973974890': 2016-10-20 14:34:04,294 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' released barrier for procedure'rolllog', counting down latch. Waiting for 1 more 2016-10-20 14:34:04,295 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:04,295 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:04,295 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs$1(100): Ignoring created notification for node:/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:04,295 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:04,295 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:34:04,295 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:34:04,296 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:34:04,296 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:34:04,297 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:34:04,297 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:04,298 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:04,298 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:34:04,299 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:34:04,299 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:04,300 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:04,301 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'rolllog' member 'cn012.l42scl.hortonworks.com,41289,1476973974467': 2016-10-20 14:34:04,301 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' released barrier for procedure'rolllog', counting down latch. Waiting for 0 more 2016-10-20 14:34:04,301 INFO [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(221): Procedure 'rolllog' execution completed 2016-10-20 14:34:04,301 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(230): Running finish phase. 2016-10-20 14:34:04,301 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2016-10-20 14:34:04,301 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:rolllog 2016-10-20 14:34:04,301 INFO [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure rolllogincluding nodes /1/rolllog-proc/acquired /1/rolllog-proc/reached /1/rolllog-proc/abort 2016-10-20 14:34:04,311 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/abort/rolllog 2016-10-20 14:34:04,311 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/1/rolllog-proc/abort/rolllog 2016-10-20 14:34:04,311 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/abort/rolllog 2016-10-20 14:34:04,312 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(80): Received created event:/1/rolllog-proc/abort/rolllog 2016-10-20 14:34:04,312 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/rolllog-proc/abort/rolllog 2016-10-20 14:34:04,312 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/rolllog-proc/abort/rolllog 2016-10-20 14:34:04,313 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:04,313 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /1/rolllog-proc/abort/rolllog 2016-10-20 14:34:04,313 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2016-10-20 14:34:04,314 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/1/rolllog-proc 2016-10-20 14:34:04,313 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/abort 2016-10-20 14:34:04,314 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/rolllog-proc/abort 2016-10-20 14:34:04,314 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/rolllog-proc/abort' 2016-10-20 14:34:04,314 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:04,314 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2016-10-20 14:34:04,315 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(317): Aborting procedure member for znode /1/rolllog-proc/abort/rolllog 2016-10-20 14:34:04,315 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:34:04,315 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2016-10-20 14:34:04,316 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:34:04,317 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:04,317 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:04,317 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:04,318 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] zookeeper.ZKUtil(365): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on existing znode=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:04,318 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2016-10-20 14:34:04,318 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----rolllog 2016-10-20 14:34:04,319 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:04,320 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:04,328 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/acquired 2016-10-20 14:34:04,329 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/rolllog-proc/acquired 2016-10-20 14:34:04,329 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/rolllog-proc/acquired' 2016-10-20 14:34:04,329 DEBUG [(cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2016-10-20 14:34:04,329 INFO [B.defaultRpcServer.handler=4,queue=0,port=41289] master.LogRollMasterProcedureManager(116): Done waiting - exec procedure for rolllog 2016-10-20 14:34:04,332 INFO [B.defaultRpcServer.handler=4,queue=0,port=41289] master.LogRollMasterProcedureManager(117): Distributed roll log procedure is successful! 2016-10-20 14:34:04,329 DEBUG [main-EventThread] zookeeper.ZKUtil(624): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Unable to get data of znode /1/rolllog-proc/abort/rolllog because node does not exist (not an error) 2016-10-20 14:34:04,332 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/abort 2016-10-20 14:34:04,332 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/abort 2016-10-20 14:34:04,333 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/rolllog-proc/abort 2016-10-20 14:34:04,333 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/rolllog-proc/abort' 2016-10-20 14:34:04,333 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(110): Received procedure abort children changed event: /1/rolllog-proc/abort 2016-10-20 14:34:04,333 DEBUG [main] client.HBaseAdmin(2516): Waiting a max of 300000 ms for procedure 'rolllog-proc : rolllog'' to complete. (max 857 ms per retry) 2016-10-20 14:34:04,333 DEBUG [main] client.HBaseAdmin(2525): (#1) Sleeping: 100ms while waiting for procedure completion. 2016-10-20 14:34:04,333 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/1/rolllog-proc/abort' 2016-10-20 14:34:04,333 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:04,334 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog 2016-10-20 14:34:04,334 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:04,334 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/acquired/rolllog 2016-10-20 14:34:04,334 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/acquired 2016-10-20 14:34:04,335 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(107): Received procedure start children changed event: /1/rolllog-proc/acquired 2016-10-20 14:34:04,335 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/1/rolllog-proc/acquired' 2016-10-20 14:34:04,335 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:04,335 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog 2016-10-20 14:34:04,336 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:04,336 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/reached/rolllog 2016-10-20 14:34:04,336 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rolllog-proc/abort/rolllog 2016-10-20 14:34:04,434 DEBUG [main] client.HBaseAdmin(2531): Getting current status of procedure from master... 2016-10-20 14:34:04,436 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(904): Checking to see if procedure from request:rolllog-proc is done 2016-10-20 14:34:04,436 DEBUG [main] impl.BackupSystemTable(222): read region server last roll log result to hbase:backup 2016-10-20 14:34:04,443 DEBUG [main] impl.IncrementalBackupManager(214): In getLogFilesForNewBackup() olderTimestamps: {cn012.l42scl.hortonworks.com:41289=1476974022000, cn012.l42scl.hortonworks.com:37560=1476974021985} newestTimestamps: {cn012.l42scl.hortonworks.com:41289=1476974036670, cn012.l42scl.hortonworks.com:37560=1476974037083} 2016-10-20 14:34:04,447 DEBUG [main] impl.IncrementalBackupManager(260): currentLogFile: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974036606 2016-10-20 14:34:04,448 DEBUG [main] impl.IncrementalBackupManager(260): currentLogFile: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974042857 2016-10-20 14:34:04,448 DEBUG [main] impl.IncrementalBackupManager(260): currentLogFile: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974037083 2016-10-20 14:34:04,448 DEBUG [main] impl.IncrementalBackupManager(260): currentLogFile: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974043279 2016-10-20 14:34:04,450 DEBUG [main] impl.IncrementalBackupManager$NewestLogFilter(341): Skip .meta log file: cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.meta.1476973975915.meta 2016-10-20 14:34:04,450 DEBUG [main] impl.IncrementalBackupManager(260): currentLogFile: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974042856 2016-10-20 14:34:04,450 DEBUG [main] impl.IncrementalBackupManager(260): currentLogFile: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974043284 2016-10-20 14:34:04,452 DEBUG [main] impl.BackupSystemTable(619): get WAL files from hbase:backup 2016-10-20 14:34:04,481 INFO [IPC Server handler 6 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741893_1069{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:34:04,485 DEBUG [main] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/backupUT/backup_1476974042777/ns2/test-14769740038901/.tabledesc/.tableinfo.0000000001 2016-10-20 14:34:04,486 DEBUG [main] util.BackupServerUtil(182): Attempting to copy table info for:ns2:test-14769740038901 target: hdfs://localhost:37879/backupUT/backup_1476974042777/ns2/test-14769740038901 descriptor: TableDescriptor{hTableDescriptor='ns2:test-14769740038901', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}} 2016-10-20 14:34:04,486 DEBUG [main] util.BackupServerUtil(184): Finished copying tableinfo. 2016-10-20 14:34:04,493 DEBUG [main] util.BackupServerUtil(188): Starting to write region info for table ns2:test-14769740038901 2016-10-20 14:34:04,501 INFO [IPC Server handler 0 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741894_1070{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:34:04,506 DEBUG [main] util.BackupServerUtil(197): Finished writing region info for table ns2:test-14769740038901 2016-10-20 14:34:04,524 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741895_1071{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:34:04,528 DEBUG [main] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/backupUT/backup_1476974042777/ns1/test-1476974003890/.tabledesc/.tableinfo.0000000001 2016-10-20 14:34:04,529 DEBUG [main] util.BackupServerUtil(182): Attempting to copy table info for:ns1:test-1476974003890 target: hdfs://localhost:37879/backupUT/backup_1476974042777/ns1/test-1476974003890 descriptor: TableDescriptor{hTableDescriptor='ns1:test-1476974003890', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}} 2016-10-20 14:34:04,529 DEBUG [main] util.BackupServerUtil(184): Finished copying tableinfo. 2016-10-20 14:34:04,535 DEBUG [main] util.BackupServerUtil(188): Starting to write region info for table ns1:test-1476974003890 2016-10-20 14:34:04,544 INFO [IPC Server handler 1 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741896_1072{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 49 2016-10-20 14:34:04,949 DEBUG [main] util.BackupServerUtil(197): Finished writing region info for table ns1:test-1476974003890 2016-10-20 14:34:04,949 INFO [main] impl.IncrementalTableBackupClient(107): Incremental copy is starting. 2016-10-20 14:34:04,957 DEBUG [main] mapreduce.MapReduceBackupCopyTask(304): Doing COPY_TYPE_DISTCP 2016-10-20 14:34:05,000 DEBUG [main] mapreduce.MapReduceBackupCopyTask(313): DistCp options: [hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974036606, hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974037083, hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974036606, hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974036670, hdfs://localhost:37879/backupUT/backup_1476974042777/WALs] 2016-10-20 14:34:05,089 WARN [main] mapreduce.JobResourceUploader(64): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-10-20 14:34:05,318 INFO [IPC Server handler 5 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741897_1073{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 1734 2016-10-20 14:34:05,753 INFO [IPC Server handler 3 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741898_1074{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:34:05,782 INFO [IPC Server handler 7 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741899_1075{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:34:05,812 INFO [IPC Server handler 6 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741900_1076{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:34:06,259 INFO [main] mapreduce.MapReduceBackupCopyTask$BackupDistCp(244): Progress: 100.0% subTask: 1.0 mapProgress: 1.0 2016-10-20 14:34:06,260 DEBUG [main] impl.BackupSystemTable(123): update backup status in hbase:backup for: backup_1476974042777 set status=RUNNING 2016-10-20 14:34:06,266 DEBUG [main] mapreduce.MapReduceBackupCopyTask(137): Backup progress data "100%" has been updated to hbase:backup for backup_1476974042777 2016-10-20 14:34:06,266 DEBUG [main] mapreduce.MapReduceBackupCopyTask$BackupDistCp(253): Backup progress data updated to hbase:backup: "Progress: 100.0% - 7155 bytes copied." 2016-10-20 14:34:06,266 DEBUG [main] mapreduce.MapReduceBackupCopyTask$BackupDistCp(267): DistCp job-id: job_local1123199809_0004 completed: true true 2016-10-20 14:34:06,286 DEBUG [main] mapreduce.MapReduceBackupCopyTask$BackupDistCp(270): Counters: 23 File System Counters FILE: Number of bytes read=48235659 FILE: Number of bytes written=48010030 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=123173 HDFS: Number of bytes written=2241159 HDFS: Number of read operations=703 HDFS: Number of large read operations=0 HDFS: Number of write operations=297 Map-Reduce Framework Map input records=4 Map output records=0 Input split bytes=257 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=0 Total committed heap usage (bytes)=2457337856 File Input Format Counters Bytes Read=1566 File Output Format Counters Bytes Written=0 org.apache.hadoop.tools.mapred.CopyMapper$Counter BYTESCOPIED=7155 BYTESEXPECTED=7155 COPY=4 2016-10-20 14:34:06,293 INFO [main] impl.IncrementalTableBackupClient(157): Incremental copy from hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974036606,hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974037083,hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974036606,hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974036670 to hdfs://localhost:37879/backupUT/backup_1476974042777/WALs finished. 2016-10-20 14:34:06,293 DEBUG [main] impl.BackupSystemTable(599): add WAL files to hbase:backup: backup_1476974042777 hdfs://localhost:37879/backupUT files [hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974036606,hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974037083,hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974036606,hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974036670] 2016-10-20 14:34:06,293 DEBUG [main] impl.BackupSystemTable(602): add :hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974036606 2016-10-20 14:34:06,293 DEBUG [main] impl.BackupSystemTable(602): add :hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974037083 2016-10-20 14:34:06,293 DEBUG [main] impl.BackupSystemTable(602): add :hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974036606 2016-10-20 14:34:06,294 DEBUG [main] impl.BackupSystemTable(602): add :hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974036670 2016-10-20 14:34:06,299 DEBUG [main] impl.BackupSystemTable(464): read RS log ts from hbase:backup for root=hdfs://localhost:37879/backupUT 2016-10-20 14:34:06,305 DEBUG [main] impl.BackupSystemTable(436): write RS log time stamps to hbase:backup for tables [ns2:test-14769740038901,ns1:test-1476974003890] 2016-10-20 14:34:06,310 DEBUG [main] impl.BackupSystemTable(464): read RS log ts from hbase:backup for root=hdfs://localhost:37879/backupUT 2016-10-20 14:34:06,315 DEBUG [main] impl.BackupSystemTable(205): write backup start code to hbase:backup 1476974036670 2016-10-20 14:34:06,319 DEBUG [main] impl.BackupManifest(477): 1 tables exist in table set. 2016-10-20 14:34:06,319 DEBUG [main] impl.BackupManager(304): Getting the direct ancestors of the current backup backup_1476974042777 2016-10-20 14:34:06,319 DEBUG [main] impl.BackupSystemTable(271): get backup history from hbase:backup 2016-10-20 14:34:06,319 DEBUG [main] impl.BackupSystemTable(404): get backup contexts from hbase:backup 2016-10-20 14:34:06,324 DEBUG [main] impl.BackupManager(348): Current backup has an incremental backup ancestor, touching its image manifest in hdfs://localhost:37879/backupUT/backup_1476974036506/WALs to construct the dependency. 2016-10-20 14:34:06,325 DEBUG [main] impl.BackupManifest(348): Loading manifest from: hdfs://localhost:37879/backupUT/backup_1476974036506/WALs 2016-10-20 14:34:06,331 DEBUG [main] impl.BackupManifest(402): Loaded manifest instance from manifest file: /backupUT/backup_1476974036506/WALs/.backup.manifest 2016-10-20 14:34:06,331 DEBUG [main] impl.BackupManager(355): Last dependent incremental backup image information: 2016-10-20 14:34:06,331 DEBUG [main] impl.BackupManager(356): Token: backup_1476974021410 2016-10-20 14:34:06,332 DEBUG [main] impl.BackupManager(357): Backup directory: hdfs://localhost:37879/backupUT 2016-10-20 14:34:06,332 DEBUG [main] impl.BackupManifest(777): Full image set can cover image backup_1476974021410 2016-10-20 14:34:06,332 DEBUG [main] impl.BackupManager(361): Got 1 ancestors for the current backup. 2016-10-20 14:34:06,332 DEBUG [main] impl.BackupManifest(607): hdfs://localhost:37879/backupUT backup_1476974042777 INCREMENTAL 2016-10-20 14:34:06,332 DEBUG [main] impl.BackupManager(304): Getting the direct ancestors of the current backup backup_1476974042777 2016-10-20 14:34:06,332 DEBUG [main] impl.BackupSystemTable(271): get backup history from hbase:backup 2016-10-20 14:34:06,333 DEBUG [main] impl.BackupSystemTable(404): get backup contexts from hbase:backup 2016-10-20 14:34:06,337 DEBUG [main] impl.BackupManager(348): Current backup has an incremental backup ancestor, touching its image manifest in hdfs://localhost:37879/backupUT/backup_1476974036506/WALs to construct the dependency. 2016-10-20 14:34:06,338 DEBUG [main] impl.BackupManifest(348): Loading manifest from: hdfs://localhost:37879/backupUT/backup_1476974036506/WALs 2016-10-20 14:34:06,342 DEBUG [main] impl.BackupManifest(402): Loaded manifest instance from manifest file: /backupUT/backup_1476974036506/WALs/.backup.manifest 2016-10-20 14:34:06,343 DEBUG [main] impl.BackupManager(355): Last dependent incremental backup image information: 2016-10-20 14:34:06,343 DEBUG [main] impl.BackupManager(356): Token: backup_1476974021410 2016-10-20 14:34:06,343 DEBUG [main] impl.BackupManager(357): Backup directory: hdfs://localhost:37879/backupUT 2016-10-20 14:34:06,343 DEBUG [main] impl.BackupManifest(777): Full image set can cover image backup_1476974021410 2016-10-20 14:34:06,343 DEBUG [main] impl.BackupManager(361): Got 1 ancestors for the current backup. 2016-10-20 14:34:06,352 INFO [IPC Server handler 5 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741901_1077{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 634 2016-10-20 14:34:06,757 INFO [main] impl.BackupManifest(512): Manifest file stored to hdfs://localhost:37879/backupUT/backup_1476974042777/ns2/test-14769740038901/.backup.manifest 2016-10-20 14:34:06,758 DEBUG [main] impl.BackupManifest(477): 1 tables exist in table set. 2016-10-20 14:34:06,758 DEBUG [main] impl.BackupManager(304): Getting the direct ancestors of the current backup backup_1476974042777 2016-10-20 14:34:06,758 DEBUG [main] impl.BackupSystemTable(271): get backup history from hbase:backup 2016-10-20 14:34:06,758 DEBUG [main] impl.BackupSystemTable(404): get backup contexts from hbase:backup 2016-10-20 14:34:06,765 DEBUG [main] impl.BackupManager(348): Current backup has an incremental backup ancestor, touching its image manifest in hdfs://localhost:37879/backupUT/backup_1476974036506/WALs to construct the dependency. 2016-10-20 14:34:06,766 DEBUG [main] impl.BackupManifest(348): Loading manifest from: hdfs://localhost:37879/backupUT/backup_1476974036506/WALs 2016-10-20 14:34:06,773 DEBUG [main] impl.BackupManifest(402): Loaded manifest instance from manifest file: /backupUT/backup_1476974036506/WALs/.backup.manifest 2016-10-20 14:34:06,773 DEBUG [main] impl.BackupManager(355): Last dependent incremental backup image information: 2016-10-20 14:34:06,773 DEBUG [main] impl.BackupManager(356): Token: backup_1476974021410 2016-10-20 14:34:06,773 DEBUG [main] impl.BackupManager(357): Backup directory: hdfs://localhost:37879/backupUT 2016-10-20 14:34:06,773 DEBUG [main] impl.BackupManifest(777): Full image set can cover image backup_1476974021410 2016-10-20 14:34:06,774 DEBUG [main] impl.BackupManager(361): Got 1 ancestors for the current backup. 2016-10-20 14:34:06,774 DEBUG [main] impl.BackupManifest(607): hdfs://localhost:37879/backupUT backup_1476974042777 INCREMENTAL 2016-10-20 14:34:06,774 DEBUG [main] impl.BackupManager(304): Getting the direct ancestors of the current backup backup_1476974042777 2016-10-20 14:34:06,774 DEBUG [main] impl.BackupSystemTable(271): get backup history from hbase:backup 2016-10-20 14:34:06,774 DEBUG [main] impl.BackupSystemTable(404): get backup contexts from hbase:backup 2016-10-20 14:34:06,781 DEBUG [main] impl.BackupManager(348): Current backup has an incremental backup ancestor, touching its image manifest in hdfs://localhost:37879/backupUT/backup_1476974036506/WALs to construct the dependency. 2016-10-20 14:34:06,781 DEBUG [main] impl.BackupManifest(348): Loading manifest from: hdfs://localhost:37879/backupUT/backup_1476974036506/WALs 2016-10-20 14:34:06,787 DEBUG [main] impl.BackupManifest(402): Loaded manifest instance from manifest file: /backupUT/backup_1476974036506/WALs/.backup.manifest 2016-10-20 14:34:06,787 DEBUG [main] impl.BackupManager(355): Last dependent incremental backup image information: 2016-10-20 14:34:06,787 DEBUG [main] impl.BackupManager(356): Token: backup_1476974021410 2016-10-20 14:34:06,787 DEBUG [main] impl.BackupManager(357): Backup directory: hdfs://localhost:37879/backupUT 2016-10-20 14:34:06,788 DEBUG [main] impl.BackupManifest(777): Full image set can cover image backup_1476974021410 2016-10-20 14:34:06,788 DEBUG [main] impl.BackupManager(361): Got 1 ancestors for the current backup. 2016-10-20 14:34:06,798 INFO [IPC Server handler 9 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741902_1078{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:34:06,799 INFO [main] impl.BackupManifest(512): Manifest file stored to hdfs://localhost:37879/backupUT/backup_1476974042777/ns1/test-1476974003890/.backup.manifest 2016-10-20 14:34:06,799 DEBUG [main] impl.BackupManifest(477): 2 tables exist in table set. 2016-10-20 14:34:06,799 DEBUG [main] impl.BackupManager(304): Getting the direct ancestors of the current backup backup_1476974042777 2016-10-20 14:34:06,799 DEBUG [main] impl.BackupSystemTable(271): get backup history from hbase:backup 2016-10-20 14:34:06,800 DEBUG [main] impl.BackupSystemTable(404): get backup contexts from hbase:backup 2016-10-20 14:34:06,805 DEBUG [main] impl.BackupManager(348): Current backup has an incremental backup ancestor, touching its image manifest in hdfs://localhost:37879/backupUT/backup_1476974036506/WALs to construct the dependency. 2016-10-20 14:34:06,806 DEBUG [main] impl.BackupManifest(348): Loading manifest from: hdfs://localhost:37879/backupUT/backup_1476974036506/WALs 2016-10-20 14:34:06,811 DEBUG [main] impl.BackupManifest(402): Loaded manifest instance from manifest file: /backupUT/backup_1476974036506/WALs/.backup.manifest 2016-10-20 14:34:06,811 DEBUG [main] impl.BackupManager(355): Last dependent incremental backup image information: 2016-10-20 14:34:06,811 DEBUG [main] impl.BackupManager(356): Token: backup_1476974021410 2016-10-20 14:34:06,811 DEBUG [main] impl.BackupManager(357): Backup directory: hdfs://localhost:37879/backupUT 2016-10-20 14:34:06,812 DEBUG [main] impl.BackupManifest(777): Full image set can cover image backup_1476974021410 2016-10-20 14:34:06,812 DEBUG [main] impl.BackupManager(361): Got 1 ancestors for the current backup. 2016-10-20 14:34:06,812 DEBUG [main] impl.BackupManifest(607): hdfs://localhost:37879/backupUT backup_1476974042777 INCREMENTAL 2016-10-20 14:34:06,821 INFO [IPC Server handler 0 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741903_1079{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:34:06,822 INFO [main] impl.BackupManifest(512): Manifest file stored to hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/.backup.manifest 2016-10-20 14:34:06,823 DEBUG [main] impl.FullTableBackupClient(408): in-fly convert code here, provided by future jira 2016-10-20 14:34:06,823 DEBUG [main] impl.FullTableBackupClient(416): Backup backup_1476974042777 finished: type=INCREMENTAL,tablelist=ns2:test-14769740038901;ns1:test-1476974003890,targetRootDir=hdfs://localhost:37879/backupUT,startts=1476974042786,completets=1476974046319,bytescopied=0 2016-10-20 14:34:06,823 DEBUG [main] impl.BackupSystemTable(123): update backup status in hbase:backup for: backup_1476974042777 set status=COMPLETE 2016-10-20 14:34:06,829 INFO [main] impl.FullTableBackupClient(431): Backup backup_1476974042777 completed. 2016-10-20 14:34:06,829 DEBUG [main] impl.BackupSystemTable(157): read backup status from hbase:backup for: backup_1476974042777 2016-10-20 14:34:06,831 DEBUG [main] backup.TestIncrementalBackup(166): Restoring full backup_1476974021410 2016-10-20 14:34:06,839 INFO [main] impl.RestoreTablesClient(95): HBase table ns1:table1_restore does not exist. It will be created during restore process 2016-10-20 14:34:06,841 INFO [main] impl.RestoreTablesClient(95): HBase table ns2:table2_restore does not exist. It will be created during restore process 2016-10-20 14:34:06,842 DEBUG [main] impl.BackupManifest(348): Loading manifest from: hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/.backup.manifest 2016-10-20 14:34:06,847 DEBUG [main] impl.BackupManifest(402): Loaded manifest instance from manifest file: /backupUT/backup_1476974021410/ns1/test-1476974003890/.backup.manifest 2016-10-20 14:34:06,848 DEBUG [main] impl.BackupManifest(348): Loading manifest from: hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901/.backup.manifest 2016-10-20 14:34:06,852 DEBUG [main] impl.BackupManifest(402): Loaded manifest instance from manifest file: /backupUT/backup_1476974021410/ns2/test-14769740038901/.backup.manifest 2016-10-20 14:34:06,855 DEBUG [main] impl.BackupManifest(348): Loading manifest from: hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/.backup.manifest 2016-10-20 14:34:06,859 DEBUG [main] impl.BackupManifest(402): Loaded manifest instance from manifest file: /backupUT/backup_1476974021410/ns1/test-1476974003890/.backup.manifest 2016-10-20 14:34:06,859 INFO [main] impl.RestoreTablesClient(145): Restoring 'ns1:test-1476974003890' to 'ns1:table1_restore' from full backup image hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890 2016-10-20 14:34:06,883 DEBUG [main] util.RestoreServerUtil(400): obtained descriptor from snapshot for ns1:test-1476974003890 2016-10-20 14:34:06,891 INFO [main] util.RestoreServerUtil(707): Creating target table 'ns1:table1_restore' 2016-10-20 14:34:06,891 DEBUG [main] util.RestoreServerUtil(609): Parsing region dir: hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/archive/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246 2016-10-20 14:34:06,894 DEBUG [main] util.RestoreServerUtil(639): Parsing family dir [hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/archive/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f in region [hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/archive/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246] 2016-10-20 14:34:06,895 INFO [main] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=3, currentSize=1080968, freeSize=1042881336, maxSize=1043962304, heapSize=1080968, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:06,902 DEBUG [main] util.RestoreServerUtil(659): Trying to figure out region boundaries hfile=hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/archive/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49 first=row0 last=row98 2016-10-20 14:34:06,902 DEBUG [main] util.RestoreServerUtil(639): Parsing family dir [hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/archive/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f3 in region [hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/archive/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246] 2016-10-20 14:34:06,904 INFO [main] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=3, currentSize=1080968, freeSize=1042881336, maxSize=1043962304, heapSize=1080968, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:06,909 DEBUG [main] util.RestoreServerUtil(659): Trying to figure out region boundaries hfile=hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/archive/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339 first=row-ns1:test-1476974003890-3-0 last=row-ns1:test-1476974003890-3-5 2016-10-20 14:34:06,919 INFO [B.defaultRpcServer.handler=3,queue=0,port=41289] master.HMaster(1501): Client=hbase//172.18.128.12 create 'ns1:table1_restore', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f3', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2016-10-20 14:34:07,037 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-jobhistoryserver.properties,hadoop-metrics2.properties 2016-10-20 14:34:07,066 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure CreateTableProcedure (table=ns1:table1_restore) id=15 owner=hbase state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2016-10-20 14:34:07,077 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=15 2016-10-20 14:34:07,105 DEBUG [ProcedureExecutor-14] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/ns1:table1_restore/write-master:412890000000000 2016-10-20 14:34:07,179 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=15 2016-10-20 14:34:07,279 INFO [IPC Server handler 1 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741904_1080{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:34:07,284 DEBUG [ProcedureExecutor-14] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/.tabledesc/.tableinfo.0000000001 2016-10-20 14:34:07,287 INFO [RegionOpenAndInitThread-ns1:table1_restore-1] regionserver.HRegion(6162): creating HRegion ns1:table1_restore HTD == 'ns1:table1_restore', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f3', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp Table name == ns1:table1_restore 2016-10-20 14:34:07,287 INFO [RegionOpenAndInitThread-ns1:table1_restore-2] regionserver.HRegion(6162): creating HRegion ns1:table1_restore HTD == 'ns1:table1_restore', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f3', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp Table name == ns1:table1_restore 2016-10-20 14:34:07,305 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741906_1082{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:34:07,308 DEBUG [RegionOpenAndInitThread-ns1:table1_restore-2] regionserver.HRegion(736): Instantiated ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:07,310 DEBUG [RegionOpenAndInitThread-ns1:table1_restore-2] regionserver.HRegion(1419): Closing ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.: disabling compactions & flushes 2016-10-20 14:34:07,311 DEBUG [RegionOpenAndInitThread-ns1:table1_restore-2] regionserver.HRegion(1446): Updates disabled for region ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:07,311 INFO [RegionOpenAndInitThread-ns1:table1_restore-2] regionserver.HRegion(1552): Closed ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:07,311 INFO [IPC Server handler 3 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741905_1081{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 49 2016-10-20 14:34:07,381 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=15 2016-10-20 14:34:07,684 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=15 2016-10-20 14:34:07,712 DEBUG [RegionOpenAndInitThread-ns1:table1_restore-1] regionserver.HRegion(736): Instantiated ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:07,713 DEBUG [RegionOpenAndInitThread-ns1:table1_restore-1] regionserver.HRegion(1419): Closing ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.: disabling compactions & flushes 2016-10-20 14:34:07,713 DEBUG [RegionOpenAndInitThread-ns1:table1_restore-1] regionserver.HRegion(1446): Updates disabled for region ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:07,713 INFO [RegionOpenAndInitThread-ns1:table1_restore-1] regionserver.HRegion(1552): Closed ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:07,869 DEBUG [ProcedureExecutor-14] hbase.MetaTableAccessor(1374): Put{"totalColumns":1,"row":"ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.","families":{"info":[{"qualifier":"regioninfo","vlen":48,"tag":[],"timestamp":9223372036854775807}]}}, Put{"totalColumns":1,"row":"ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.","families":{"info":[{"qualifier":"regioninfo","vlen":48,"tag":[],"timestamp":9223372036854775807}]}} 2016-10-20 14:34:07,873 INFO [ProcedureExecutor-14] hbase.MetaTableAccessor(1571): Added 2 2016-10-20 14:34:08,030 INFO [ProcedureExecutor-14] master.AssignmentManager(726): Assigning 2 region(s) to cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:08,031 ERROR [ProcedureExecutor-14] master.TableStateManager(134): Unable to get table ns1:table1_restore state org.apache.hadoop.hbase.TableNotFoundException: ns1:table1_restore at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546) at org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494) 2016-10-20 14:34:08,032 ERROR [ProcedureExecutor-14] master.TableStateManager(134): Unable to get table ns1:table1_restore state org.apache.hadoop.hbase.TableNotFoundException: ns1:table1_restore at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546) at org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494) 2016-10-20 14:34:08,032 INFO [ProcedureExecutor-14] master.RegionStates(1106): Transition {a1135d09b318d77673368ed569f0b066 state=OFFLINE, ts=1476974048027, server=null} to {a1135d09b318d77673368ed569f0b066 state=PENDING_OPEN, ts=1476974048032, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:08,033 INFO [ProcedureExecutor-14] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. with state=PENDING_OPEN, sn=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:08,034 INFO [ProcedureExecutor-14] master.RegionStates(1106): Transition {b35704e7c3c2eb86497b08776362fa99 state=OFFLINE, ts=1476974048029, server=null} to {b35704e7c3c2eb86497b08776362fa99 state=PENDING_OPEN, ts=1476974048034, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:08,034 INFO [ProcedureExecutor-14] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. with state=PENDING_OPEN, sn=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:08,037 INFO [PriorityRpcServer.handler=2,queue=0,port=37560] regionserver.RSRpcServices(1666): Open ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:08,042 INFO [PriorityRpcServer.handler=2,queue=0,port=37560] regionserver.RSRpcServices(1666): Open ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:08,044 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(6339): Opening region: {ENCODED => a1135d09b318d77673368ed569f0b066, NAME => 'ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.', STARTKEY => 'row0', ENDKEY => ''} 2016-10-20 14:34:08,047 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table table1_restore a1135d09b318d77673368ed569f0b066 2016-10-20 14:34:08,047 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(6339): Opening region: {ENCODED => b35704e7c3c2eb86497b08776362fa99, NAME => 'ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.', STARTKEY => '', ENDKEY => 'row0'} 2016-10-20 14:34:08,047 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(736): Instantiated ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:08,048 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table table1_restore b35704e7c3c2eb86497b08776362fa99 2016-10-20 14:34:08,049 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(736): Instantiated ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:08,052 INFO [StoreOpener-a1135d09b318d77673368ed569f0b066-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=3, currentSize=1080968, freeSize=1042881336, maxSize=1043962304, heapSize=1080968, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:08,053 INFO [StoreOpener-a1135d09b318d77673368ed569f0b066-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:08,056 INFO [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=3, currentSize=1080968, freeSize=1042881336, maxSize=1043962304, heapSize=1080968, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:08,057 DEBUG [StoreOpener-a1135d09b318d77673368ed569f0b066-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f 2016-10-20 14:34:08,057 INFO [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:08,059 DEBUG [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f 2016-10-20 14:34:08,059 INFO [StoreOpener-a1135d09b318d77673368ed569f0b066-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=3, currentSize=1080968, freeSize=1042881336, maxSize=1043962304, heapSize=1080968, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:08,059 INFO [StoreOpener-a1135d09b318d77673368ed569f0b066-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:08,060 DEBUG [StoreOpener-a1135d09b318d77673368ed569f0b066-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f3 2016-10-20 14:34:08,063 INFO [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=3, currentSize=1080968, freeSize=1042881336, maxSize=1043962304, heapSize=1080968, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:08,063 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066 2016-10-20 14:34:08,064 INFO [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:08,065 DEBUG [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f3 2016-10-20 14:34:08,066 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.FlushLargeStoresPolicy(72): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table ns1:table1_restore, use config (67108864) instead 2016-10-20 14:34:08,066 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99 2016-10-20 14:34:08,070 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.FlushLargeStoresPolicy(72): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table ns1:table1_restore, use config (67108864) instead 2016-10-20 14:34:08,070 DEBUG [RS_OPEN_REGION-cn012:37560-1] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:34:08,070 INFO [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(871): Onlined a1135d09b318d77673368ed569f0b066; next sequenceid=2 2016-10-20 14:34:08,085 INFO [PostOpenDeployTasks:a1135d09b318d77673368ed569f0b066] regionserver.HRegionServer(1957): Post open deploy tasks for ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:08,086 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.AssignmentManager(2884): Got transition OPENED for {a1135d09b318d77673368ed569f0b066 state=PENDING_OPEN, ts=1476974048032, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:08,086 INFO [B.defaultRpcServer.handler=3,queue=0,port=41289] master.RegionStates(1106): Transition {a1135d09b318d77673368ed569f0b066 state=PENDING_OPEN, ts=1476974048032, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {a1135d09b318d77673368ed569f0b066 state=OPEN, ts=1476974048086, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:08,087 INFO [B.defaultRpcServer.handler=3,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. with state=OPEN, openSeqNum=2, server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:08,088 DEBUG [RS_OPEN_REGION-cn012:37560-2] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:34:08,088 INFO [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(871): Onlined b35704e7c3c2eb86497b08776362fa99; next sequenceid=2 2016-10-20 14:34:08,088 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.RegionStates(452): Onlined a1135d09b318d77673368ed569f0b066 on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:08,089 ERROR [B.defaultRpcServer.handler=3,queue=0,port=41289] master.TableStateManager(134): Unable to get table ns1:table1_restore state org.apache.hadoop.hbase.TableNotFoundException: ns1:table1_restore at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionOpen(AssignmentManager.java:2311) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionTransition(AssignmentManager.java:2891) at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1350) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8623) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2229) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:136) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:111) at java.lang.Thread.run(Thread.java:745) 2016-10-20 14:34:08,090 INFO [PostOpenDeployTasks:b35704e7c3c2eb86497b08776362fa99] regionserver.HRegionServer(1957): Post open deploy tasks for ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:08,090 DEBUG [PostOpenDeployTasks:a1135d09b318d77673368ed569f0b066] regionserver.HRegionServer(1984): Finished post open deploy task for ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:08,099 DEBUG [RS_OPEN_REGION-cn012:37560-1] handler.OpenRegionHandler(126): Opened ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:08,100 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.AssignmentManager(2884): Got transition OPENED for {b35704e7c3c2eb86497b08776362fa99 state=PENDING_OPEN, ts=1476974048034, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:08,100 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.RegionStates(1106): Transition {b35704e7c3c2eb86497b08776362fa99 state=PENDING_OPEN, ts=1476974048034, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {b35704e7c3c2eb86497b08776362fa99 state=OPEN, ts=1476974048100, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:08,100 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. with state=OPEN, openSeqNum=2, server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:08,101 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.RegionStates(452): Onlined b35704e7c3c2eb86497b08776362fa99 on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:08,102 DEBUG [ProcedureExecutor-14] master.AssignmentManager(897): Bulk assigning done for cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:08,102 DEBUG [ProcedureExecutor-14] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"ns1:table1_restore","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476974048102}]}} 2016-10-20 14:34:08,102 ERROR [B.defaultRpcServer.handler=0,queue=0,port=41289] master.TableStateManager(134): Unable to get table ns1:table1_restore state org.apache.hadoop.hbase.TableNotFoundException: ns1:table1_restore at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionOpen(AssignmentManager.java:2311) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionTransition(AssignmentManager.java:2891) at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1350) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8623) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2229) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:136) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:111) at java.lang.Thread.run(Thread.java:745) 2016-10-20 14:34:08,103 DEBUG [PostOpenDeployTasks:b35704e7c3c2eb86497b08776362fa99] regionserver.HRegionServer(1984): Finished post open deploy task for ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:08,103 DEBUG [RS_OPEN_REGION-cn012:37560-2] handler.OpenRegionHandler(126): Opened ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:08,104 INFO [ProcedureExecutor-14] hbase.MetaTableAccessor(1700): Updated table ns1:table1_restore state to ENABLED in META 2016-10-20 14:34:08,186 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=15 2016-10-20 14:34:08,620 DEBUG [ProcedureExecutor-14] lock.ZKInterProcessLockBase(328): Released /1/table-lock/ns1:table1_restore/write-master:412890000000000 2016-10-20 14:34:08,620 DEBUG [ProcedureExecutor-14] procedure2.ProcedureExecutor(870): Procedure completed in 1.5370sec: CreateTableProcedure (table=ns1:table1_restore) id=15 owner=hbase state=FINISHED 2016-10-20 14:34:09,188 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=15 2016-10-20 14:34:09,189 INFO [main] client.HBaseAdmin$TableFuture(3337): Operation: CREATE, Table Name: ns1:table1_restore completed 2016-10-20 14:34:09,199 DEBUG [main] client.ConnectionImplementation(604): Table ns1:table1_restore should be available 2016-10-20 14:34:09,204 DEBUG [main] util.RestoreServerUtil(339): cluster hold the backup image: hdfs://localhost:37879; local cluster node: hdfs://localhost:37879 2016-10-20 14:34:09,204 DEBUG [main] util.RestoreServerUtil(345): File hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/archive/data/ns1/test-1476974003890 on local cluster, back it up before restore 2016-10-20 14:34:09,233 INFO [IPC Server handler 1 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741907_1083{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 8292 2016-10-20 14:34:09,652 INFO [IPC Server handler 4 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741908_1084{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:34:09,653 DEBUG [main] util.RestoreServerUtil(355): Copied to temporary path on local cluster: /user/hbase/hbase-staging/restore 2016-10-20 14:34:09,655 DEBUG [main] util.RestoreServerUtil(463): TableArchivePath for bulkload using tempPath: /user/hbase/hbase-staging/restore 2016-10-20 14:34:09,662 INFO [main] util.RestoreServerUtil(578): Setting configuration for restore with LoadIncrementalHFile: hbase.rpc.timeout to 2 minutes, to handle the number of files in backup /user/hbase/hbase-staging/restore 2016-10-20 14:34:09,695 DEBUG [main] util.RestoreServerUtil(472): Restoring HFiles from directory hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246 2016-10-20 14:34:09,696 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x5e26f1ed connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:34:09,720 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x5e26f1ed0x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:34:09,726 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39666e42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:34:09,726 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x5e26f1ed-0x157e2823ec60010 connected 2016-10-20 14:34:09,726 DEBUG [main] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:34:09,727 DEBUG [main] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:34:09,740 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42874; # active connections: 8 2016-10-20 14:34:09,740 DEBUG [AsyncRpcChannel-pool2-t19] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:09,744 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:09,744 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42874 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:09,761 DEBUG [main] client.ConnectionImplementation(604): Table ns1:table1_restore should be available 2016-10-20 14:34:09,782 DEBUG [AsyncRpcChannel-pool2-t20] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service MasterService, sasl=false 2016-10-20 14:34:09,782 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42876; # active connections: 9 2016-10-20 14:34:09,785 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:09,785 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42876 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:09,811 INFO [LoadIncrementalHFiles-0] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=3, currentSize=1080968, freeSize=1042881336, maxSize=1043962304, heapSize=1080968, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:09,811 INFO [LoadIncrementalHFiles-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=3, currentSize=1080968, freeSize=1042881336, maxSize=1043962304, heapSize=1080968, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:09,818 INFO [LoadIncrementalHFiles-0] mapreduce.LoadIncrementalHFiles(820): Trying to load hfile=hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49 first=row0 last=row98 2016-10-20 14:34:09,818 INFO [LoadIncrementalHFiles-1] mapreduce.LoadIncrementalHFiles(820): Trying to load hfile=hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339 first=row-ns1:test-1476974003890-3-0 last=row-ns1:test-1476974003890-3-5 2016-10-20 14:34:09,838 DEBUG [LoadIncrementalHFiles-2] mapreduce.LoadIncrementalHFiles$4(913): Going to connect to server region=ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99., hostname=cn012.l42scl.hortonworks.com,37560,1476973974890, seqNum=2 for row with hfile group [{[B@5cd96b41,hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339}] 2016-10-20 14:34:09,841 DEBUG [LoadIncrementalHFiles-3] mapreduce.LoadIncrementalHFiles$4(913): Going to connect to server region=ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066., hostname=cn012.l42scl.hortonworks.com,37560,1476973974890, seqNum=2 for row row0 with hfile group [{[B@4cc65c2,hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49}] 2016-10-20 14:34:09,857 DEBUG [AsyncRpcChannel-pool2-t21] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:09,857 DEBUG [RpcServer.listener,port=37560] ipc.RpcServer$Listener(880): RpcServer.listener,port=37560: connection from 172.18.128.12:42250; # active connections: 7 2016-10-20 14:34:09,858 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:09,859 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42250 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:09,860 INFO [B.defaultRpcServer.handler=1,queue=0,port=37560] regionserver.HStore(670): Validating hfile at hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339 for inclusion in store f3 region ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:09,860 INFO [B.defaultRpcServer.handler=2,queue=0,port=37560] regionserver.HStore(670): Validating hfile at hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49 for inclusion in store f region ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:09,866 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=37560] regionserver.HStore(682): HFile bounds: first=row0 last=row98 2016-10-20 14:34:09,866 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=37560] regionserver.HStore(684): Region bounds: first=row0 last= 2016-10-20 14:34:09,866 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=37560] regionserver.HStore(682): HFile bounds: first=row-ns1:test-1476974003890-3-0 last=row-ns1:test-1476974003890-3-5 2016-10-20 14:34:09,866 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=37560] regionserver.HStore(684): Region bounds: first= last=row0 2016-10-20 14:34:09,874 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=37560] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49 as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f/0d8044dfb2744f219ae918e8da007398_SeqId_4_ 2016-10-20 14:34:09,874 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=37560] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339 as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f3/ec4fb540c34b489cae106284681a22bd_SeqId_4_ 2016-10-20 14:34:09,875 INFO [B.defaultRpcServer.handler=2,queue=0,port=37560] regionserver.HStore(742): Loaded HFile hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49 into store 'f' as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f/0d8044dfb2744f219ae918e8da007398_SeqId_4_ - updating store file list. 2016-10-20 14:34:09,876 INFO [B.defaultRpcServer.handler=1,queue=0,port=37560] regionserver.HStore(742): Loaded HFile hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339 into store 'f3' as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f3/ec4fb540c34b489cae106284681a22bd_SeqId_4_ - updating store file list. 2016-10-20 14:34:09,884 INFO [B.defaultRpcServer.handler=1,queue=0,port=37560] regionserver.HStore(777): Loaded HFile hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f3/ec4fb540c34b489cae106284681a22bd_SeqId_4_ into store 'f3 2016-10-20 14:34:09,884 INFO [B.defaultRpcServer.handler=2,queue=0,port=37560] regionserver.HStore(777): Loaded HFile hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f/0d8044dfb2744f219ae918e8da007398_SeqId_4_ into store 'f 2016-10-20 14:34:09,884 INFO [B.defaultRpcServer.handler=1,queue=0,port=37560] regionserver.HStore(748): Successfully loaded store file hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339 into store f3 (new location: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f3/ec4fb540c34b489cae106284681a22bd_SeqId_4_) 2016-10-20 14:34:09,884 INFO [B.defaultRpcServer.handler=2,queue=0,port=37560] regionserver.HStore(748): Successfully loaded store file hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49 into store f (new location: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f/0d8044dfb2744f219ae918e8da007398_SeqId_4_) 2016-10-20 14:34:09,893 INFO [main] client.ConnectionImplementation(1805): Closing master protocol: MasterService 2016-10-20 14:34:09,894 INFO [main] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec60010 2016-10-20 14:34:09,903 DEBUG [main] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:09,906 INFO [main] impl.RestoreTablesClient(205): Restore includes the following image(s): 2016-10-20 14:34:09,906 INFO [main] impl.RestoreTablesClient(207): Backup: backup_1476974021410 hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/ 2016-10-20 14:34:09,906 DEBUG [AsyncRpcChannel-pool2-t19] ipc.AsyncRpcChannel$8(566): IPC Client (-1015703562) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:09,906 DEBUG [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42874 because read count=-1. Number of active connections: 9 2016-10-20 14:34:09,906 DEBUG [AsyncRpcChannel-pool2-t21] ipc.AsyncRpcChannel$8(566): IPC Client (-105687103) to cn012.l42scl.hortonworks.com/172.18.128.12:37560 from hbase: closed 2016-10-20 14:34:09,906 DEBUG [AsyncRpcChannel-pool2-t20] ipc.AsyncRpcChannel$8(566): IPC Client (1817876032) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:09,906 DEBUG [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Listener(912): RpcServer.listener,port=37560: DISCONNECTING client 172.18.128.12:42250 because read count=-1. Number of active connections: 7 2016-10-20 14:34:09,906 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42876 because read count=-1. Number of active connections: 9 2016-10-20 14:34:09,912 DEBUG [main] impl.BackupManifest(348): Loading manifest from: hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901/.backup.manifest 2016-10-20 14:34:09,917 DEBUG [main] impl.BackupManifest(402): Loaded manifest instance from manifest file: /backupUT/backup_1476974021410/ns2/test-14769740038901/.backup.manifest 2016-10-20 14:34:09,917 INFO [main] impl.RestoreTablesClient(145): Restoring 'ns2:test-14769740038901' to 'ns2:table2_restore' from full backup image hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901 2016-10-20 14:34:09,931 DEBUG [main] util.RestoreServerUtil(400): obtained descriptor from snapshot for ns2:test-14769740038901 2016-10-20 14:34:09,938 INFO [main] util.RestoreServerUtil(707): Creating target table 'ns2:table2_restore' 2016-10-20 14:34:09,938 DEBUG [main] util.RestoreServerUtil(609): Parsing region dir: hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901/archive/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391 2016-10-20 14:34:09,940 DEBUG [main] util.RestoreServerUtil(639): Parsing family dir [hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901/archive/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/f in region [hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901/archive/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391] 2016-10-20 14:34:09,941 INFO [main] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=3, currentSize=1080968, freeSize=1042881336, maxSize=1043962304, heapSize=1080968, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:09,946 DEBUG [main] util.RestoreServerUtil(659): Trying to figure out region boundaries hfile=hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901/archive/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/f/a39b76fbbc004d008400803cfd71b6e0 first=row0 last=row98 2016-10-20 14:34:09,948 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.HMaster(1501): Client=hbase//172.18.128.12 create 'ns2:table2_restore', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2016-10-20 14:34:10,098 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure CreateTableProcedure (table=ns2:table2_restore) id=16 owner=hbase state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2016-10-20 14:34:10,106 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=16 2016-10-20 14:34:10,162 DEBUG [ProcedureExecutor-15] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/ns2:table2_restore/write-master:412890000000000 2016-10-20 14:34:10,208 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=16 2016-10-20 14:34:10,335 INFO [IPC Server handler 6 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741909_1085{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 290 2016-10-20 14:34:10,410 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=16 2016-10-20 14:34:10,713 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=16 2016-10-20 14:34:10,741 DEBUG [ProcedureExecutor-15] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns2/table2_restore/.tabledesc/.tableinfo.0000000001 2016-10-20 14:34:10,744 INFO [RegionOpenAndInitThread-ns2:table2_restore-1] regionserver.HRegion(6162): creating HRegion ns2:table2_restore HTD == 'ns2:table2_restore', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp Table name == ns2:table2_restore 2016-10-20 14:34:10,757 INFO [IPC Server handler 5 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741910_1086{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:34:10,758 DEBUG [RegionOpenAndInitThread-ns2:table2_restore-1] regionserver.HRegion(736): Instantiated ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:10,762 DEBUG [RegionOpenAndInitThread-ns2:table2_restore-1] regionserver.HRegion(1419): Closing ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8.: disabling compactions & flushes 2016-10-20 14:34:10,762 DEBUG [RegionOpenAndInitThread-ns2:table2_restore-1] regionserver.HRegion(1446): Updates disabled for region ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:10,762 INFO [RegionOpenAndInitThread-ns2:table2_restore-1] regionserver.HRegion(1552): Closed ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:10,916 DEBUG [ProcedureExecutor-15] hbase.MetaTableAccessor(1374): Put{"totalColumns":1,"row":"ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8.","families":{"info":[{"qualifier":"regioninfo","vlen":44,"tag":[],"timestamp":9223372036854775807}]}} 2016-10-20 14:34:10,925 INFO [ProcedureExecutor-15] hbase.MetaTableAccessor(1571): Added 1 2016-10-20 14:34:11,067 INFO [ProcedureExecutor-15] master.AssignmentManager(726): Assigning 1 region(s) to cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:11,068 ERROR [ProcedureExecutor-15] master.TableStateManager(134): Unable to get table ns2:table2_restore state org.apache.hadoop.hbase.TableNotFoundException: ns2:table2_restore at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546) at org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:127) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:57) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494) 2016-10-20 14:34:11,068 INFO [ProcedureExecutor-15] master.RegionStates(1106): Transition {81900e9262bb3c77f4b3fd7b6387aeb8 state=OFFLINE, ts=1476974051067, server=null} to {81900e9262bb3c77f4b3fd7b6387aeb8 state=PENDING_OPEN, ts=1476974051068, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:11,069 INFO [ProcedureExecutor-15] master.RegionStateStore(207): Updating hbase:meta row ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. with state=PENDING_OPEN, sn=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:11,072 INFO [PriorityRpcServer.handler=3,queue=1,port=37560] regionserver.RSRpcServices(1666): Open ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:11,082 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(6339): Opening region: {ENCODED => 81900e9262bb3c77f4b3fd7b6387aeb8, NAME => 'ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8.', STARTKEY => '', ENDKEY => ''} 2016-10-20 14:34:11,082 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table table2_restore 81900e9262bb3c77f4b3fd7b6387aeb8 2016-10-20 14:34:11,083 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(736): Instantiated ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:11,089 INFO [StoreOpener-81900e9262bb3c77f4b3fd7b6387aeb8-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=3, currentSize=1080968, freeSize=1042881336, maxSize=1043962304, heapSize=1080968, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:11,089 INFO [StoreOpener-81900e9262bb3c77f4b3fd7b6387aeb8-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:11,091 DEBUG [StoreOpener-81900e9262bb3c77f4b3fd7b6387aeb8-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/f 2016-10-20 14:34:11,093 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8 2016-10-20 14:34:11,103 DEBUG [RS_OPEN_REGION-cn012:37560-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:34:11,103 INFO [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(871): Onlined 81900e9262bb3c77f4b3fd7b6387aeb8; next sequenceid=2 2016-10-20 14:34:11,105 INFO [PostOpenDeployTasks:81900e9262bb3c77f4b3fd7b6387aeb8] regionserver.HRegionServer(1957): Post open deploy tasks for ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:11,107 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.AssignmentManager(2884): Got transition OPENED for {81900e9262bb3c77f4b3fd7b6387aeb8 state=PENDING_OPEN, ts=1476974051068, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:11,107 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.RegionStates(1106): Transition {81900e9262bb3c77f4b3fd7b6387aeb8 state=PENDING_OPEN, ts=1476974051068, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {81900e9262bb3c77f4b3fd7b6387aeb8 state=OPEN, ts=1476974051107, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:11,107 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. with state=OPEN, openSeqNum=2, server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:11,109 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.RegionStates(452): Onlined 81900e9262bb3c77f4b3fd7b6387aeb8 on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:11,109 DEBUG [ProcedureExecutor-15] master.AssignmentManager(897): Bulk assigning done for cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:11,110 DEBUG [ProcedureExecutor-15] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"ns2:table2_restore","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476974051110}]}} 2016-10-20 14:34:11,110 ERROR [B.defaultRpcServer.handler=0,queue=0,port=41289] master.TableStateManager(134): Unable to get table ns2:table2_restore state org.apache.hadoop.hbase.TableNotFoundException: ns2:table2_restore at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionOpen(AssignmentManager.java:2311) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionTransition(AssignmentManager.java:2891) at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1350) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8623) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2229) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:136) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:111) at java.lang.Thread.run(Thread.java:745) 2016-10-20 14:34:11,111 DEBUG [PostOpenDeployTasks:81900e9262bb3c77f4b3fd7b6387aeb8] regionserver.HRegionServer(1984): Finished post open deploy task for ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:11,115 DEBUG [RS_OPEN_REGION-cn012:37560-0] handler.OpenRegionHandler(126): Opened ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:11,116 INFO [ProcedureExecutor-15] hbase.MetaTableAccessor(1700): Updated table ns2:table2_restore state to ENABLED in META 2016-10-20 14:34:11,215 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=16 2016-10-20 14:34:11,595 DEBUG [ProcedureExecutor-15] lock.ZKInterProcessLockBase(328): Released /1/table-lock/ns2:table2_restore/write-master:412890000000000 2016-10-20 14:34:11,595 DEBUG [ProcedureExecutor-15] procedure2.ProcedureExecutor(870): Procedure completed in 1.4930sec: CreateTableProcedure (table=ns2:table2_restore) id=16 owner=hbase state=FINISHED 2016-10-20 14:34:12,217 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=16 2016-10-20 14:34:12,218 INFO [main] client.HBaseAdmin$TableFuture(3337): Operation: CREATE, Table Name: ns2:table2_restore completed 2016-10-20 14:34:12,230 DEBUG [main] client.ConnectionImplementation(604): Table ns2:table2_restore should be available 2016-10-20 14:34:12,235 DEBUG [main] util.RestoreServerUtil(339): cluster hold the backup image: hdfs://localhost:37879; local cluster node: hdfs://localhost:37879 2016-10-20 14:34:12,235 DEBUG [main] util.RestoreServerUtil(345): File hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901/archive/data/ns2/test-14769740038901 on local cluster, back it up before restore 2016-10-20 14:34:12,265 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741911_1087{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 8292 2016-10-20 14:34:12,667 DEBUG [main] util.RestoreServerUtil(355): Copied to temporary path on local cluster: /user/hbase/hbase-staging/restore 2016-10-20 14:34:12,669 DEBUG [main] util.RestoreServerUtil(463): TableArchivePath for bulkload using tempPath: /user/hbase/hbase-staging/restore 2016-10-20 14:34:12,710 DEBUG [main] util.RestoreServerUtil(472): Restoring HFiles from directory hdfs://localhost:37879/user/hbase/hbase-staging/restore/20fb71680e16fdda348e8695b59a8391 2016-10-20 14:34:12,711 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0xec7b5de connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:34:12,728 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0xec7b5de0x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:34:12,742 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0xec7b5de-0x157e2823ec60011 connected 2016-10-20 14:34:12,742 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@373e6b9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:34:12,743 DEBUG [main] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:34:12,743 DEBUG [main] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:34:12,749 DEBUG [AsyncRpcChannel-pool2-t22] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:12,749 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42902; # active connections: 8 2016-10-20 14:34:12,752 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:12,753 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42902 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:12,769 DEBUG [main] client.ConnectionImplementation(604): Table ns2:table2_restore should be available 2016-10-20 14:34:12,784 DEBUG [AsyncRpcChannel-pool2-t23] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service MasterService, sasl=false 2016-10-20 14:34:12,784 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42904; # active connections: 9 2016-10-20 14:34:12,787 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:12,787 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42904 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:12,797 INFO [LoadIncrementalHFiles-0] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=3, currentSize=1080968, freeSize=1042881336, maxSize=1043962304, heapSize=1080968, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:12,804 INFO [LoadIncrementalHFiles-0] mapreduce.LoadIncrementalHFiles(820): Trying to load hfile=hdfs://localhost:37879/user/hbase/hbase-staging/restore/20fb71680e16fdda348e8695b59a8391/f/a39b76fbbc004d008400803cfd71b6e0 first=row0 last=row98 2016-10-20 14:34:12,813 DEBUG [LoadIncrementalHFiles-1] mapreduce.LoadIncrementalHFiles$4(913): Going to connect to server region=ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8., hostname=cn012.l42scl.hortonworks.com,37560,1476973974890, seqNum=2 for row with hfile group [{[B@7412a438,hdfs://localhost:37879/user/hbase/hbase-staging/restore/20fb71680e16fdda348e8695b59a8391/f/a39b76fbbc004d008400803cfd71b6e0}] 2016-10-20 14:34:12,819 DEBUG [AsyncRpcChannel-pool2-t24] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:12,819 DEBUG [RpcServer.listener,port=37560] ipc.RpcServer$Listener(880): RpcServer.listener,port=37560: connection from 172.18.128.12:42274; # active connections: 7 2016-10-20 14:34:12,822 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:12,822 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42274 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:12,823 INFO [B.defaultRpcServer.handler=3,queue=0,port=37560] regionserver.HStore(670): Validating hfile at hdfs://localhost:37879/user/hbase/hbase-staging/restore/20fb71680e16fdda348e8695b59a8391/f/a39b76fbbc004d008400803cfd71b6e0 for inclusion in store f region ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:12,829 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=37560] regionserver.HStore(682): HFile bounds: first=row0 last=row98 2016-10-20 14:34:12,829 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=37560] regionserver.HStore(684): Region bounds: first= last= 2016-10-20 14:34:12,833 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=37560] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/hbase-staging/restore/20fb71680e16fdda348e8695b59a8391/f/a39b76fbbc004d008400803cfd71b6e0 as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/f/f821ebe52bba4f08b45eba7f62321282_SeqId_4_ 2016-10-20 14:34:12,835 INFO [B.defaultRpcServer.handler=3,queue=0,port=37560] regionserver.HStore(742): Loaded HFile hdfs://localhost:37879/user/hbase/hbase-staging/restore/20fb71680e16fdda348e8695b59a8391/f/a39b76fbbc004d008400803cfd71b6e0 into store 'f' as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/f/f821ebe52bba4f08b45eba7f62321282_SeqId_4_ - updating store file list. 2016-10-20 14:34:12,845 INFO [B.defaultRpcServer.handler=3,queue=0,port=37560] regionserver.HStore(777): Loaded HFile hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/f/f821ebe52bba4f08b45eba7f62321282_SeqId_4_ into store 'f 2016-10-20 14:34:12,845 INFO [B.defaultRpcServer.handler=3,queue=0,port=37560] regionserver.HStore(748): Successfully loaded store file hdfs://localhost:37879/user/hbase/hbase-staging/restore/20fb71680e16fdda348e8695b59a8391/f/a39b76fbbc004d008400803cfd71b6e0 into store f (new location: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/f/f821ebe52bba4f08b45eba7f62321282_SeqId_4_) 2016-10-20 14:34:12,848 INFO [main] client.ConnectionImplementation(1805): Closing master protocol: MasterService 2016-10-20 14:34:12,848 INFO [main] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec60011 2016-10-20 14:34:12,853 DEBUG [main] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:12,855 INFO [main] impl.RestoreTablesClient(205): Restore includes the following image(s): 2016-10-20 14:34:12,855 DEBUG [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Listener(912): RpcServer.listener,port=37560: DISCONNECTING client 172.18.128.12:42274 because read count=-1. Number of active connections: 7 2016-10-20 14:34:12,855 DEBUG [AsyncRpcChannel-pool2-t23] ipc.AsyncRpcChannel$8(566): IPC Client (811795741) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:12,856 DEBUG [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42902 because read count=-1. Number of active connections: 9 2016-10-20 14:34:12,855 DEBUG [AsyncRpcChannel-pool2-t24] ipc.AsyncRpcChannel$8(566): IPC Client (1834038667) to cn012.l42scl.hortonworks.com/172.18.128.12:37560 from hbase: closed 2016-10-20 14:34:12,855 DEBUG [AsyncRpcChannel-pool2-t22] ipc.AsyncRpcChannel$8(566): IPC Client (-1536318528) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:12,855 DEBUG [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42904 because read count=-1. Number of active connections: 9 2016-10-20 14:34:12,855 INFO [main] impl.RestoreTablesClient(207): Backup: backup_1476974021410 hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901/ 2016-10-20 14:34:12,857 DEBUG [main] impl.RestoreTablesClient(219): restoreStage finished 2016-10-20 14:34:12,879 DEBUG [main] client.ClientScanner(279): Advancing internal scanner to startKey at 'row0' 2016-10-20 14:34:12,925 DEBUG [main] impl.BackupManifest(348): Loading manifest from: hdfs://localhost:37879/backupUT/backup_1476974042777/ns1/test-1476974003890/.backup.manifest 2016-10-20 14:34:12,930 DEBUG [main] impl.BackupManifest(402): Loaded manifest instance from manifest file: /backupUT/backup_1476974042777/ns1/test-1476974003890/.backup.manifest 2016-10-20 14:34:12,931 DEBUG [main] impl.BackupManifest(348): Loading manifest from: hdfs://localhost:37879/backupUT/backup_1476974042777/ns2/test-14769740038901/.backup.manifest 2016-10-20 14:34:12,934 DEBUG [main] impl.BackupManifest(402): Loaded manifest instance from manifest file: /backupUT/backup_1476974042777/ns2/test-14769740038901/.backup.manifest 2016-10-20 14:34:12,935 DEBUG [main] impl.BackupManifest(348): Loading manifest from: hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/.backup.manifest 2016-10-20 14:34:12,938 DEBUG [main] impl.BackupManifest(402): Loaded manifest instance from manifest file: /backupUT/backup_1476974021410/ns1/test-1476974003890/.backup.manifest 2016-10-20 14:34:12,938 INFO [main] impl.RestoreTablesClient(145): Restoring 'ns1:test-1476974003890' to 'ns1:table1_restore' from full backup image hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890 2016-10-20 14:34:12,941 DEBUG [main] util.RestoreServerUtil(383): Retrieved descriptor: 'ns1:test-1476974003890', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} thru backup_1476974042777 2016-10-20 14:34:12,945 INFO [main] util.RestoreServerUtil(696): Truncating exising target table 'ns1:table1_restore', preserving region splits 2016-10-20 14:34:12,947 INFO [main] client.HBaseAdmin$10(773): Started disable of ns1:table1_restore 2016-10-20 14:34:12,951 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.HMaster(1992): Client=hbase//172.18.128.12 disable ns1:table1_restore 2016-10-20 14:34:13,109 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure DisableTableProcedure (table=ns1:table1_restore) id=17 owner=hbase state=RUNNABLE:DISABLE_TABLE_PREPARE added to the store. 2016-10-20 14:34:13,117 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=17 2016-10-20 14:34:13,130 DEBUG [ProcedureExecutor-17] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/ns1:table1_restore/write-master:412890000000001 2016-10-20 14:34:13,219 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=17 2016-10-20 14:34:13,417 DEBUG [ProcedureExecutor-17] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"ns1:table1_restore","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476974053417}]}} 2016-10-20 14:34:13,421 INFO [ProcedureExecutor-17] hbase.MetaTableAccessor(1700): Updated table ns1:table1_restore state to DISABLING in META 2016-10-20 14:34:13,422 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=17 2016-10-20 14:34:13,555 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-jobhistoryserver.properties,hadoop-metrics2.properties 2016-10-20 14:34:13,567 INFO [ProcedureExecutor-17] procedure.DisableTableProcedure(395): Offlining 2 regions. 2016-10-20 14:34:13,575 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.AssignmentManager(1352): Starting unassign of ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. (offlining), current state: {b35704e7c3c2eb86497b08776362fa99 state=OPEN, ts=1476974048100, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:13,576 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-1] master.AssignmentManager(1352): Starting unassign of ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. (offlining), current state: {a1135d09b318d77673368ed569f0b066 state=OPEN, ts=1476974048086, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:13,576 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.RegionStates(1106): Transition {b35704e7c3c2eb86497b08776362fa99 state=OPEN, ts=1476974048100, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {b35704e7c3c2eb86497b08776362fa99 state=PENDING_CLOSE, ts=1476974053576, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:13,576 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-1] master.RegionStates(1106): Transition {a1135d09b318d77673368ed569f0b066 state=OPEN, ts=1476974048086, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {a1135d09b318d77673368ed569f0b066 state=PENDING_CLOSE, ts=1476974053576, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:13,576 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. with state=PENDING_CLOSE 2016-10-20 14:34:13,576 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-1] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. with state=PENDING_CLOSE 2016-10-20 14:34:13,579 INFO [PriorityRpcServer.handler=1,queue=1,port=37560] regionserver.RSRpcServices(1314): Close a1135d09b318d77673368ed569f0b066, moving to null 2016-10-20 14:34:13,579 INFO [PriorityRpcServer.handler=4,queue=0,port=37560] regionserver.RSRpcServices(1314): Close b35704e7c3c2eb86497b08776362fa99, moving to null 2016-10-20 14:34:13,580 DEBUG [RS_CLOSE_REGION-cn012:37560-0] handler.CloseRegionHandler(90): Processing close of ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:13,580 DEBUG [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(1419): Closing ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.: disabling compactions & flushes 2016-10-20 14:34:13,580 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.AssignmentManager(930): Sent CLOSE to cn012.l42scl.hortonworks.com,37560,1476973974890 for region ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:13,580 DEBUG [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(1446): Updates disabled for region ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:13,601 INFO [StoreCloserThread-ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.-1] regionserver.HStore(839): Closed f 2016-10-20 14:34:13,601 DEBUG [RS_CLOSE_REGION-cn012:37560-2] handler.CloseRegionHandler(90): Processing close of ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:13,601 DEBUG [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1419): Closing ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.: disabling compactions & flushes 2016-10-20 14:34:13,602 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-1] master.AssignmentManager(930): Sent CLOSE to cn012.l42scl.hortonworks.com,37560,1476973974890 for region ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:13,602 DEBUG [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1446): Updates disabled for region ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:13,604 INFO [StoreCloserThread-ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.-1] regionserver.HStore(839): Closed f3 2016-10-20 14:34:13,628 INFO [StoreCloserThread-ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.-1] regionserver.HStore(839): Closed f 2016-10-20 14:34:13,628 INFO [StoreCloserThread-ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.-1] regionserver.HStore(839): Closed f3 2016-10-20 14:34:13,629 DEBUG [RS_CLOSE_REGION-cn012:37560-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/recovered.edits/6.seqid to file, newSeqId=6, maxSeqId=2 2016-10-20 14:34:13,631 INFO [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(1552): Closed ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:13,632 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.AssignmentManager(2884): Got transition CLOSED for {b35704e7c3c2eb86497b08776362fa99 state=PENDING_CLOSE, ts=1476974053576, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:13,633 INFO [B.defaultRpcServer.handler=2,queue=0,port=41289] master.RegionStates(1106): Transition {b35704e7c3c2eb86497b08776362fa99 state=PENDING_CLOSE, ts=1476974053576, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {b35704e7c3c2eb86497b08776362fa99 state=OFFLINE, ts=1476974053633, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:13,633 INFO [B.defaultRpcServer.handler=2,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. with state=OFFLINE 2016-10-20 14:34:13,636 INFO [B.defaultRpcServer.handler=2,queue=0,port=41289] master.RegionStates(590): Offlined b35704e7c3c2eb86497b08776362fa99 from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:13,636 DEBUG [RS_CLOSE_REGION-cn012:37560-0] handler.CloseRegionHandler(122): Closed ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:13,649 DEBUG [RS_CLOSE_REGION-cn012:37560-2] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/recovered.edits/6.seqid to file, newSeqId=6, maxSeqId=2 2016-10-20 14:34:13,650 INFO [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1552): Closed ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:13,651 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.AssignmentManager(2884): Got transition CLOSED for {a1135d09b318d77673368ed569f0b066 state=PENDING_CLOSE, ts=1476974053576, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:13,652 INFO [B.defaultRpcServer.handler=3,queue=0,port=41289] master.RegionStates(1106): Transition {a1135d09b318d77673368ed569f0b066 state=PENDING_CLOSE, ts=1476974053576, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {a1135d09b318d77673368ed569f0b066 state=OFFLINE, ts=1476974053652, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:13,652 INFO [B.defaultRpcServer.handler=3,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. with state=OFFLINE 2016-10-20 14:34:13,654 INFO [B.defaultRpcServer.handler=3,queue=0,port=41289] master.RegionStates(590): Offlined a1135d09b318d77673368ed569f0b066 from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:13,654 DEBUG [RS_CLOSE_REGION-cn012:37560-2] handler.CloseRegionHandler(122): Closed ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:13,724 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=17 2016-10-20 14:34:13,825 DEBUG [ProcedureExecutor-17] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"ns1:table1_restore","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476974053824}]}} 2016-10-20 14:34:13,828 INFO [ProcedureExecutor-17] hbase.MetaTableAccessor(1700): Updated table ns1:table1_restore state to DISABLED in META 2016-10-20 14:34:13,828 INFO [ProcedureExecutor-17] procedure.DisableTableProcedure(424): Disabled table, ns1:table1_restore, is completed. 2016-10-20 14:34:14,120 DEBUG [ProcedureExecutor-17] lock.ZKInterProcessLockBase(328): Released /1/table-lock/ns1:table1_restore/write-master:412890000000001 2016-10-20 14:34:14,120 DEBUG [ProcedureExecutor-17] procedure2.ProcedureExecutor(870): Procedure completed in 1.0060sec: DisableTableProcedure (table=ns1:table1_restore) id=17 owner=hbase state=FINISHED 2016-10-20 14:34:14,226 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=17 2016-10-20 14:34:14,227 INFO [main] client.HBaseAdmin$TableFuture(3337): Operation: DISABLE, Table Name: ns1:table1_restore completed 2016-10-20 14:34:14,229 INFO [main] client.HBaseAdmin$8(608): Started truncating ns1:table1_restore 2016-10-20 14:34:14,236 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.HMaster(1854): Client=hbase//172.18.128.12 truncate ns1:table1_restore 2016-10-20 14:34:14,393 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure TruncateTableProcedure (table=ns1:table1_restore preserveSplits=true) id=18 owner=hbase state=RUNNABLE:TRUNCATE_TABLE_PRE_OPERATION added to the store. 2016-10-20 14:34:14,413 DEBUG [ProcedureExecutor-16] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/ns1:table1_restore/write-master:412890000000002 2016-10-20 14:34:14,416 DEBUG [ProcedureExecutor-16] procedure.TruncateTableProcedure(87): waiting for 'ns1:table1_restore' regions in transition 2016-10-20 14:34:14,571 DEBUG [ProcedureExecutor-16] hbase.MetaTableAccessor(1406): Delete{"totalColumns":1,"row":"ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1476974054569}]},"ts":9223372036854775807}, Delete{"totalColumns":1,"row":"ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1476974054569}]},"ts":9223372036854775807} 2016-10-20 14:34:14,575 INFO [ProcedureExecutor-16] hbase.MetaTableAccessor(1854): Deleted [{ENCODED => b35704e7c3c2eb86497b08776362fa99, NAME => 'ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.', STARTKEY => '', ENDKEY => 'row0'}, {ENCODED => a1135d09b318d77673368ed569f0b066, NAME => 'ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.', STARTKEY => 'row0', ENDKEY => ''}] 2016-10-20 14:34:14,582 DEBUG [ProcedureExecutor-16] procedure.DeleteTableProcedure(408): Removing 'ns1:table1_restore' from region states. 2016-10-20 14:34:14,588 DEBUG [ProcedureExecutor-16] procedure.DeleteTableProcedure(412): Marking 'ns1:table1_restore' as deleted. 2016-10-20 14:34:14,588 DEBUG [ProcedureExecutor-16] hbase.MetaTableAccessor(1406): Delete{"totalColumns":1,"row":"ns1:table1_restore","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":1476974054588}]},"ts":9223372036854775807} 2016-10-20 14:34:14,591 INFO [ProcedureExecutor-16] hbase.MetaTableAccessor(1726): Deleted table ns1:table1_restore state from META 2016-10-20 14:34:14,739 DEBUG [ProcedureExecutor-16] procedure.DeleteTableProcedure(340): Archiving region ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. from FS 2016-10-20 14:34:14,739 DEBUG [ProcedureExecutor-16] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99 2016-10-20 14:34:14,747 DEBUG [ProcedureExecutor-16] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f3, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/recovered.edits] 2016-10-20 14:34:14,764 DEBUG [ProcedureExecutor-16] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f3/ec4fb540c34b489cae106284681a22bd_SeqId_4_, to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/archive/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f3/ec4fb540c34b489cae106284681a22bd_SeqId_4_ 2016-10-20 14:34:14,773 DEBUG [ProcedureExecutor-16] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/recovered.edits/6.seqid, to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/archive/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/recovered.edits/6.seqid 2016-10-20 14:34:14,774 INFO [IPC Server handler 3 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741905_1081 127.0.0.1:38642 2016-10-20 14:34:14,774 DEBUG [ProcedureExecutor-16] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99 2016-10-20 14:34:14,774 DEBUG [ProcedureExecutor-16] procedure.DeleteTableProcedure(340): Archiving region ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. from FS 2016-10-20 14:34:14,774 DEBUG [ProcedureExecutor-16] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066 2016-10-20 14:34:14,779 DEBUG [ProcedureExecutor-16] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f3, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/recovered.edits] 2016-10-20 14:34:14,790 DEBUG [ProcedureExecutor-16] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f/0d8044dfb2744f219ae918e8da007398_SeqId_4_, to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/archive/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f/0d8044dfb2744f219ae918e8da007398_SeqId_4_ 2016-10-20 14:34:14,800 DEBUG [ProcedureExecutor-16] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/recovered.edits/6.seqid, to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/archive/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/recovered.edits/6.seqid 2016-10-20 14:34:14,801 INFO [IPC Server handler 0 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741906_1082 127.0.0.1:38642 2016-10-20 14:34:14,802 DEBUG [ProcedureExecutor-16] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066 2016-10-20 14:34:14,802 DEBUG [ProcedureExecutor-16] procedure.DeleteTableProcedure(344): Table 'ns1:table1_restore' archived! 2016-10-20 14:34:14,804 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741904_1080 127.0.0.1:38642 2016-10-20 14:34:14,995 INFO [IPC Server handler 7 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741912_1088{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:34:14,999 DEBUG [ProcedureExecutor-16] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns1/table1_restore/.tabledesc/.tableinfo.0000000001 2016-10-20 14:34:15,002 INFO [RegionOpenAndInitThread-ns1:table1_restore-1] regionserver.HRegion(6162): creating HRegion ns1:table1_restore HTD == 'ns1:table1_restore', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f3', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp Table name == ns1:table1_restore 2016-10-20 14:34:15,003 INFO [RegionOpenAndInitThread-ns1:table1_restore-2] regionserver.HRegion(6162): creating HRegion ns1:table1_restore HTD == 'ns1:table1_restore', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f3', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp Table name == ns1:table1_restore 2016-10-20 14:34:15,021 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741913_1089{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 49 2016-10-20 14:34:15,022 INFO [IPC Server handler 5 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741914_1090{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 49 2016-10-20 14:34:15,423 DEBUG [RegionOpenAndInitThread-ns1:table1_restore-1] regionserver.HRegion(736): Instantiated ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:15,423 DEBUG [RegionOpenAndInitThread-ns1:table1_restore-1] regionserver.HRegion(1419): Closing ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.: disabling compactions & flushes 2016-10-20 14:34:15,424 DEBUG [RegionOpenAndInitThread-ns1:table1_restore-2] regionserver.HRegion(736): Instantiated ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:15,424 DEBUG [RegionOpenAndInitThread-ns1:table1_restore-1] regionserver.HRegion(1446): Updates disabled for region ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:15,424 DEBUG [RegionOpenAndInitThread-ns1:table1_restore-2] regionserver.HRegion(1419): Closing ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.: disabling compactions & flushes 2016-10-20 14:34:15,424 INFO [RegionOpenAndInitThread-ns1:table1_restore-1] regionserver.HRegion(1552): Closed ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:15,424 DEBUG [RegionOpenAndInitThread-ns1:table1_restore-2] regionserver.HRegion(1446): Updates disabled for region ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:15,425 INFO [RegionOpenAndInitThread-ns1:table1_restore-2] regionserver.HRegion(1552): Closed ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:15,585 DEBUG [ProcedureExecutor-16] hbase.MetaTableAccessor(1374): Put{"totalColumns":1,"row":"ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.","families":{"info":[{"qualifier":"regioninfo","vlen":48,"tag":[],"timestamp":9223372036854775807}]}}, Put{"totalColumns":1,"row":"ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.","families":{"info":[{"qualifier":"regioninfo","vlen":48,"tag":[],"timestamp":9223372036854775807}]}} 2016-10-20 14:34:15,590 INFO [ProcedureExecutor-16] hbase.MetaTableAccessor(1571): Added 2 2016-10-20 14:34:15,613 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@3dddefd8] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:38642 to delete [blk_1073741904_1080, blk_1073741905_1081, blk_1073741906_1082] 2016-10-20 14:34:15,741 INFO [ProcedureExecutor-16] master.AssignmentManager(726): Assigning 2 region(s) to cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:15,742 ERROR [ProcedureExecutor-16] master.TableStateManager(134): Unable to get table ns1:table1_restore state org.apache.hadoop.hbase.TableNotFoundException: ns1:table1_restore at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546) at org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430) at org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure.executeFromState(TruncateTableProcedure.java:122) at org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure.executeFromState(TruncateTableProcedure.java:47) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494) 2016-10-20 14:34:15,743 ERROR [ProcedureExecutor-16] master.TableStateManager(134): Unable to get table ns1:table1_restore state org.apache.hadoop.hbase.TableNotFoundException: ns1:table1_restore at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546) at org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430) at org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure.executeFromState(TruncateTableProcedure.java:122) at org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure.executeFromState(TruncateTableProcedure.java:47) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494) 2016-10-20 14:34:15,756 INFO [ProcedureExecutor-16] master.RegionStates(1106): Transition {b35704e7c3c2eb86497b08776362fa99 state=OFFLINE, ts=1476974055741, server=null} to {b35704e7c3c2eb86497b08776362fa99 state=PENDING_OPEN, ts=1476974055756, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:15,757 INFO [ProcedureExecutor-16] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. with state=PENDING_OPEN, sn=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:15,759 INFO [ProcedureExecutor-16] master.RegionStates(1106): Transition {a1135d09b318d77673368ed569f0b066 state=OFFLINE, ts=1476974055741, server=null} to {a1135d09b318d77673368ed569f0b066 state=PENDING_OPEN, ts=1476974055759, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:15,759 INFO [ProcedureExecutor-16] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. with state=PENDING_OPEN, sn=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:15,762 INFO [PriorityRpcServer.handler=0,queue=0,port=37560] regionserver.RSRpcServices(1666): Open ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:15,769 INFO [PriorityRpcServer.handler=0,queue=0,port=37560] regionserver.RSRpcServices(1666): Open ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:15,772 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(6339): Opening region: {ENCODED => b35704e7c3c2eb86497b08776362fa99, NAME => 'ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.', STARTKEY => '', ENDKEY => 'row0'} 2016-10-20 14:34:15,772 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table table1_restore b35704e7c3c2eb86497b08776362fa99 2016-10-20 14:34:15,775 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(6339): Opening region: {ENCODED => a1135d09b318d77673368ed569f0b066, NAME => 'ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.', STARTKEY => 'row0', ENDKEY => ''} 2016-10-20 14:34:15,775 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(736): Instantiated ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:15,776 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table table1_restore a1135d09b318d77673368ed569f0b066 2016-10-20 14:34:15,776 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(736): Instantiated ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:15,781 INFO [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:15,781 INFO [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:15,784 INFO [StoreOpener-a1135d09b318d77673368ed569f0b066-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:15,785 INFO [StoreOpener-a1135d09b318d77673368ed569f0b066-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:15,786 DEBUG [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f 2016-10-20 14:34:15,787 DEBUG [StoreOpener-a1135d09b318d77673368ed569f0b066-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f 2016-10-20 14:34:15,789 INFO [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:15,791 INFO [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:15,791 INFO [StoreOpener-a1135d09b318d77673368ed569f0b066-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:15,792 INFO [StoreOpener-a1135d09b318d77673368ed569f0b066-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:15,793 DEBUG [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f3 2016-10-20 14:34:15,794 DEBUG [StoreOpener-a1135d09b318d77673368ed569f0b066-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f3 2016-10-20 14:34:15,794 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99 2016-10-20 14:34:15,795 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066 2016-10-20 14:34:15,798 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.FlushLargeStoresPolicy(72): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table ns1:table1_restore, use config (67108864) instead 2016-10-20 14:34:15,799 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.FlushLargeStoresPolicy(72): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table ns1:table1_restore, use config (67108864) instead 2016-10-20 14:34:15,803 DEBUG [RS_OPEN_REGION-cn012:37560-1] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:34:15,804 INFO [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(871): Onlined b35704e7c3c2eb86497b08776362fa99; next sequenceid=2 2016-10-20 14:34:15,805 DEBUG [RS_OPEN_REGION-cn012:37560-2] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:34:15,805 INFO [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(871): Onlined a1135d09b318d77673368ed569f0b066; next sequenceid=2 2016-10-20 14:34:15,806 INFO [PostOpenDeployTasks:b35704e7c3c2eb86497b08776362fa99] regionserver.HRegionServer(1957): Post open deploy tasks for ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:15,807 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.AssignmentManager(2884): Got transition OPENED for {b35704e7c3c2eb86497b08776362fa99 state=PENDING_OPEN, ts=1476974055756, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:15,807 INFO [B.defaultRpcServer.handler=2,queue=0,port=41289] master.RegionStates(1106): Transition {b35704e7c3c2eb86497b08776362fa99 state=PENDING_OPEN, ts=1476974055756, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {b35704e7c3c2eb86497b08776362fa99 state=OPEN, ts=1476974055807, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:15,808 INFO [B.defaultRpcServer.handler=2,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. with state=OPEN, openSeqNum=2, server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:15,808 INFO [PostOpenDeployTasks:a1135d09b318d77673368ed569f0b066] regionserver.HRegionServer(1957): Post open deploy tasks for ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:15,809 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.AssignmentManager(2884): Got transition OPENED for {a1135d09b318d77673368ed569f0b066 state=PENDING_OPEN, ts=1476974055759, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:15,809 INFO [B.defaultRpcServer.handler=3,queue=0,port=41289] master.RegionStates(1106): Transition {a1135d09b318d77673368ed569f0b066 state=PENDING_OPEN, ts=1476974055759, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {a1135d09b318d77673368ed569f0b066 state=OPEN, ts=1476974055809, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:15,809 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.RegionStates(452): Onlined b35704e7c3c2eb86497b08776362fa99 on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:15,810 INFO [B.defaultRpcServer.handler=3,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. with state=OPEN, openSeqNum=2, server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:15,811 ERROR [B.defaultRpcServer.handler=2,queue=0,port=41289] master.TableStateManager(134): Unable to get table ns1:table1_restore state org.apache.hadoop.hbase.TableNotFoundException: ns1:table1_restore at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionOpen(AssignmentManager.java:2311) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionTransition(AssignmentManager.java:2891) at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1350) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8623) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2229) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:136) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:111) at java.lang.Thread.run(Thread.java:745) 2016-10-20 14:34:15,811 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.RegionStates(452): Onlined a1135d09b318d77673368ed569f0b066 on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:15,811 DEBUG [ProcedureExecutor-16] master.AssignmentManager(897): Bulk assigning done for cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:15,811 DEBUG [PostOpenDeployTasks:b35704e7c3c2eb86497b08776362fa99] regionserver.HRegionServer(1984): Finished post open deploy task for ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:15,814 DEBUG [RS_OPEN_REGION-cn012:37560-1] handler.OpenRegionHandler(126): Opened ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:15,814 ERROR [B.defaultRpcServer.handler=3,queue=0,port=41289] master.TableStateManager(134): Unable to get table ns1:table1_restore state org.apache.hadoop.hbase.TableNotFoundException: ns1:table1_restore at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionOpen(AssignmentManager.java:2311) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionTransition(AssignmentManager.java:2891) at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1350) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8623) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2229) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:136) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:111) at java.lang.Thread.run(Thread.java:745) 2016-10-20 14:34:15,814 DEBUG [ProcedureExecutor-16] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"ns1:table1_restore","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476974055812}]}} 2016-10-20 14:34:15,815 DEBUG [PostOpenDeployTasks:a1135d09b318d77673368ed569f0b066] regionserver.HRegionServer(1984): Finished post open deploy task for ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:15,817 DEBUG [RS_OPEN_REGION-cn012:37560-2] handler.OpenRegionHandler(126): Opened ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:15,819 INFO [ProcedureExecutor-16] hbase.MetaTableAccessor(1700): Updated table ns1:table1_restore state to ENABLED in META 2016-10-20 14:34:15,967 DEBUG [ProcedureExecutor-16] procedure.TruncateTableProcedure(129): truncate 'ns1:table1_restore' completed 2016-10-20 14:34:16,128 DEBUG [ProcedureExecutor-16] lock.ZKInterProcessLockBase(328): Released /1/table-lock/ns1:table1_restore/write-master:412890000000002 2016-10-20 14:34:16,128 DEBUG [ProcedureExecutor-16] procedure2.ProcedureExecutor(870): Procedure completed in 1.7290sec: TruncateTableProcedure (table=ns1:table1_restore preserveSplits=true) id=18 owner=hbase state=FINISHED 2016-10-20 14:34:16,149 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=18 2016-10-20 14:34:16,150 INFO [main] client.HBaseAdmin$TableFuture(3337): Operation: TRUNCATE, Table Name: ns1:table1_restore completed 2016-10-20 14:34:16,154 DEBUG [main] util.RestoreServerUtil(339): cluster hold the backup image: hdfs://localhost:37879; local cluster node: hdfs://localhost:37879 2016-10-20 14:34:16,154 DEBUG [main] util.RestoreServerUtil(345): File hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/archive/data/ns1/test-1476974003890 on local cluster, back it up before restore 2016-10-20 14:34:16,184 INFO [IPC Server handler 3 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741915_1091{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:34:16,201 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741916_1092{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:34:16,202 DEBUG [main] util.RestoreServerUtil(355): Copied to temporary path on local cluster: /user/hbase/hbase-staging/restore 2016-10-20 14:34:16,204 DEBUG [main] util.RestoreServerUtil(463): TableArchivePath for bulkload using tempPath: /user/hbase/hbase-staging/restore 2016-10-20 14:34:16,207 DEBUG [main] mapreduce.MapReduceRestoreTask(68): Restore full backup from directory hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246 from hbase tables ,ns1:test-1476974003890 to tables ,ns1:table1_restore 2016-10-20 14:34:16,208 INFO [main] mapreduce.MapReduceRestoreTask(76): Restore ns1:test-1476974003890 into ns1:table1_restore 2016-10-20 14:34:16,214 DEBUG [main] mapreduce.HFileSplitter(125): add incremental job :/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974056208 from hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246 2016-10-20 14:34:16,216 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x134f8ef6 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:34:16,228 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x134f8ef60x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:34:16,229 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a90265a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:34:16,232 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x134f8ef6-0x157e2823ec60012 connected 2016-10-20 14:34:16,232 DEBUG [main] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:34:16,232 DEBUG [main] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:34:16,237 DEBUG [AsyncRpcChannel-pool2-t25] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service MasterService, sasl=false 2016-10-20 14:34:16,237 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42936; # active connections: 8 2016-10-20 14:34:16,240 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:16,241 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42936 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:16,249 INFO [main] mapreduce.HFileOutputFormat2(478): bulkload locality sensitive enabled 2016-10-20 14:34:16,249 INFO [main] mapreduce.HFileOutputFormat2(483): Looking up current regions for table ns1:test-1476974003890 2016-10-20 14:34:16,262 DEBUG [AsyncRpcChannel-pool2-t26] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:16,262 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42938; # active connections: 9 2016-10-20 14:34:16,265 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:16,265 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42938 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:16,273 INFO [main] mapreduce.HFileOutputFormat2(485): Configuring 1 reduce partitions to match current region count 2016-10-20 14:34:16,274 INFO [main] mapreduce.HFileOutputFormat2(378): Writing partition information to /user/hbase/hbase-staging/partitions_f5340334-0af5-40d0-933c-7d74059b9b99 2016-10-20 14:34:16,298 INFO [IPC Server handler 7 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741917_1093{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:34:16,307 WARN [main] mapreduce.TableMapReduceUtil(786): The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it. 2016-10-20 14:34:16,485 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.HConstants, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-2824109155523188165.jar 2016-10-20 14:34:17,357 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-6111409074560055759.jar 2016-10-20 14:34:17,602 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.client.Put, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-3371992891148828592.jar 2016-10-20 14:34:17,617 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.CompatibilityFactory, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-3242408414613411418.jar 2016-10-20 14:34:18,671 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-2325260339395376975.jar 2016-10-20 14:34:18,672 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.zookeeper.ZooKeeper, using jar /home/hbase/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar 2016-10-20 14:34:18,673 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class io.netty.channel.Channel, using jar /home/hbase/.m2/repository/io/netty/netty-all/4.0.30.Final/netty-all-4.0.30.Final.jar 2016-10-20 14:34:18,673 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.protobuf.Message, using jar /home/hbase/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar 2016-10-20 14:34:18,674 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.common.collect.Lists, using jar /home/hbase/.m2/repository/com/google/guava/guava/12.0.1/guava-12.0.1.jar 2016-10-20 14:34:18,675 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.htrace.Trace, using jar /home/hbase/.m2/repository/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar 2016-10-20 14:34:18,675 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.codahale.metrics.MetricRegistry, using jar /home/hbase/.m2/repository/io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.jar 2016-10-20 14:34:18,873 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-2563356837936373009.jar 2016-10-20 14:34:18,873 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.KeyValue, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-2563356837936373009.jar 2016-10-20 14:34:19,138 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-jobhistoryserver.properties,hadoop-metrics2.properties 2016-10-20 14:34:19,787 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.mapreduce.HFileInputFormat2, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-4121758217067926165.jar 2016-10-20 14:34:19,788 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-2563356837936373009.jar 2016-10-20 14:34:19,788 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.KeyValue, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-2563356837936373009.jar 2016-10-20 14:34:19,789 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-4121758217067926165.jar 2016-10-20 14:34:19,789 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.1/hadoop-mapreduce-client-core-2.7.1.jar 2016-10-20 14:34:19,789 INFO [main] mapreduce.HFileOutputFormat2(498): Incremental table ns1:test-1476974003890 output configured. 2016-10-20 14:34:19,790 INFO [main] client.ConnectionImplementation(1805): Closing master protocol: MasterService 2016-10-20 14:34:19,790 INFO [main] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec60012 2016-10-20 14:34:19,803 DEBUG [main] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:19,805 DEBUG [main] mapreduce.HFileSplitter(137): success configuring load incremental job 2016-10-20 14:34:19,806 DEBUG [AsyncRpcChannel-pool2-t25] ipc.AsyncRpcChannel$8(566): IPC Client (1983162074) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:19,806 DEBUG [AsyncRpcChannel-pool2-t26] ipc.AsyncRpcChannel$8(566): IPC Client (-1639085117) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:19,806 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.common.base.Preconditions, using jar /home/hbase/.m2/repository/com/google/guava/guava/12.0.1/guava-12.0.1.jar 2016-10-20 14:34:19,806 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42936 because read count=-1. Number of active connections: 9 2016-10-20 14:34:19,806 DEBUG [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42938 because read count=-1. Number of active connections: 9 2016-10-20 14:34:19,820 WARN [main] mapreduce.JobResourceUploader(64): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-10-20 14:34:19,840 WARN [main] mapreduce.JobResourceUploader(171): No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-10-20 14:34:21,457 INFO [LocalJobRunner Map Task Executor #0] mapreduce.HFileInputFormat2$HFileRecordReader(88): Initialize HFileRecordReader for hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246/f/44766960dbed4b82a69fd46ea16c6a49 2016-10-20 14:34:21,457 INFO [LocalJobRunner Map Task Executor #0] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:21,463 INFO [LocalJobRunner Map Task Executor #0] mapreduce.HFileInputFormat2$HFileRecordReader(103): Seeking to start 2016-10-20 14:34:21,509 INFO [LocalJobRunner Map Task Executor #0] mapreduce.HFileInputFormat2$HFileRecordReader(88): Initialize HFileRecordReader for hdfs://localhost:37879/user/hbase/hbase-staging/restore/4a7f646a7c8fc84833ef454ba337a246/f3/0c854275329043169ffbbcd8476ce339 2016-10-20 14:34:21,510 INFO [LocalJobRunner Map Task Executor #0] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:21,515 INFO [LocalJobRunner Map Task Executor #0] mapreduce.HFileInputFormat2$HFileRecordReader(103): Seeking to start 2016-10-20 14:34:21,666 INFO [pool-188-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x4f82b221 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:34:21,695 DEBUG [pool-188-thread-1-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x4f82b2210x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:34:21,701 DEBUG [pool-188-thread-1-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x4f82b221-0x157e2823ec60013 connected 2016-10-20 14:34:21,702 DEBUG [pool-188-thread-1] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e4ce56, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:34:21,702 DEBUG [pool-188-thread-1] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:34:21,702 DEBUG [pool-188-thread-1] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:34:21,713 DEBUG [AsyncRpcChannel-pool2-t27] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:21,714 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42960; # active connections: 8 2016-10-20 14:34:21,715 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:21,716 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42960 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:21,719 INFO [pool-188-thread-1] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec60013 2016-10-20 14:34:21,728 DEBUG [pool-188-thread-1] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:21,730 DEBUG [pool-188-thread-1] mapreduce.HFileOutputFormat2$1(232): first rowkey: [row-ns1:test-1476974003890-3-0] 2016-10-20 14:34:21,731 DEBUG [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42960 because read count=-1. Number of active connections: 8 2016-10-20 14:34:21,731 DEBUG [pool-188-thread-1] mapreduce.HFileOutputFormat2$1(244): use favored nodes writer: cn012.l42scl.hortonworks.com 2016-10-20 14:34:21,731 DEBUG [AsyncRpcChannel-pool2-t27] ipc.AsyncRpcChannel$8(566): IPC Client (-2102227879) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:21,732 INFO [pool-188-thread-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:21,738 INFO [pool-188-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x3cdd2f20 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:34:21,744 DEBUG [pool-188-thread-1-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x3cdd2f200x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:34:21,745 DEBUG [pool-188-thread-1] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5986cfd9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:34:21,745 DEBUG [pool-188-thread-1] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:34:21,745 DEBUG [pool-188-thread-1] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:34:21,748 DEBUG [pool-188-thread-1-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x3cdd2f20-0x157e2823ec60014 connected 2016-10-20 14:34:21,753 DEBUG [AsyncRpcChannel-pool2-t28] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:21,753 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42964; # active connections: 8 2016-10-20 14:34:21,761 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:21,761 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42964 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:21,764 INFO [pool-188-thread-1] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec60014 2016-10-20 14:34:21,769 DEBUG [pool-188-thread-1] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:21,777 DEBUG [pool-188-thread-1] mapreduce.HFileOutputFormat2$1(232): first rowkey: [row0] 2016-10-20 14:34:21,778 DEBUG [AsyncRpcChannel-pool2-t28] ipc.AsyncRpcChannel$8(566): IPC Client (-2143157152) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:21,778 DEBUG [pool-188-thread-1] mapreduce.HFileOutputFormat2$1(244): use favored nodes writer: cn012.l42scl.hortonworks.com 2016-10-20 14:34:21,778 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42964 because read count=-1. Number of active connections: 8 2016-10-20 14:34:21,778 INFO [pool-188-thread-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:21,793 WARN [Thread-1967] hdfs.DFSOutputStream$DataStreamer(1413): These favored nodes were specified but not chosen: [cn012.l42scl.hortonworks.com:37560] Specified favored nodes: [cn012.l42scl.hortonworks.com:37560] 2016-10-20 14:34:21,796 INFO [IPC Server handler 7 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741918_1094{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 8396 2016-10-20 14:34:22,202 WARN [Thread-1966] hdfs.DFSOutputStream$DataStreamer(1413): These favored nodes were specified but not chosen: [cn012.l42scl.hortonworks.com:37560] Specified favored nodes: [cn012.l42scl.hortonworks.com:37560] 2016-10-20 14:34:22,206 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741919_1095{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:34:22,281 DEBUG [main] mapreduce.MapReduceRestoreTask(93): Restoring HFiles from directory /user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974056208 2016-10-20 14:34:22,281 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x2f677247 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:34:22,295 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x2f6772470x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:34:22,298 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x2f677247-0x157e2823ec60015 connected 2016-10-20 14:34:22,298 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43f03c23, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:34:22,298 DEBUG [main] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:34:22,298 DEBUG [main] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:34:22,303 DEBUG [AsyncRpcChannel-pool2-t29] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:22,303 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42972; # active connections: 8 2016-10-20 14:34:22,306 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:22,306 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42972 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:22,321 DEBUG [main] client.ConnectionImplementation(604): Table ns1:table1_restore should be available 2016-10-20 14:34:22,323 WARN [main] mapreduce.LoadIncrementalHFiles(202): Skipping non-directory hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974056208/_SUCCESS 2016-10-20 14:34:22,339 DEBUG [AsyncRpcChannel-pool2-t30] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service MasterService, sasl=false 2016-10-20 14:34:22,339 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42976; # active connections: 9 2016-10-20 14:34:22,342 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:22,342 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42976 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:22,351 INFO [LoadIncrementalHFiles-0] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:22,351 INFO [LoadIncrementalHFiles-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:22,356 INFO [LoadIncrementalHFiles-0] mapreduce.LoadIncrementalHFiles(820): Trying to load hfile=hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974056208/f/1964b9cd70b2498ba1db9d3504b4624b first=row0 last=row98 2016-10-20 14:34:22,357 INFO [LoadIncrementalHFiles-1] mapreduce.LoadIncrementalHFiles(820): Trying to load hfile=hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974056208/f3/0d18a694af6c4b9390923404a9e115eb first=row-ns1:test-1476974003890-3-0 last=row-ns1:test-1476974003890-3-5 2016-10-20 14:34:22,368 DEBUG [LoadIncrementalHFiles-2] mapreduce.LoadIncrementalHFiles$4(913): Going to connect to server region=ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99., hostname=cn012.l42scl.hortonworks.com,37560,1476973974890, seqNum=2 for row with hfile group [{[B@47b269c4,hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974056208/f3/0d18a694af6c4b9390923404a9e115eb}] 2016-10-20 14:34:22,368 DEBUG [LoadIncrementalHFiles-3] mapreduce.LoadIncrementalHFiles$4(913): Going to connect to server region=ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066., hostname=cn012.l42scl.hortonworks.com,37560,1476973974890, seqNum=2 for row row0 with hfile group [{[B@71e7adbb,hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974056208/f/1964b9cd70b2498ba1db9d3504b4624b}] 2016-10-20 14:34:22,375 DEBUG [AsyncRpcChannel-pool2-t31] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:22,376 DEBUG [RpcServer.listener,port=37560] ipc.RpcServer$Listener(880): RpcServer.listener,port=37560: connection from 172.18.128.12:42348; # active connections: 7 2016-10-20 14:34:22,376 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:22,376 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42348 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:22,377 INFO [B.defaultRpcServer.handler=3,queue=0,port=37560] regionserver.HStore(670): Validating hfile at hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974056208/f3/0d18a694af6c4b9390923404a9e115eb for inclusion in store f3 region ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:22,377 INFO [B.defaultRpcServer.handler=4,queue=0,port=37560] regionserver.HStore(670): Validating hfile at hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974056208/f/1964b9cd70b2498ba1db9d3504b4624b for inclusion in store f region ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:22,381 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=37560] regionserver.HStore(682): HFile bounds: first=row0 last=row98 2016-10-20 14:34:22,381 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=37560] regionserver.HStore(682): HFile bounds: first=row-ns1:test-1476974003890-3-0 last=row-ns1:test-1476974003890-3-5 2016-10-20 14:34:22,381 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=37560] regionserver.HStore(684): Region bounds: first=row0 last= 2016-10-20 14:34:22,381 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=37560] regionserver.HStore(684): Region bounds: first= last=row0 2016-10-20 14:34:22,386 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=37560] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974056208/f3/0d18a694af6c4b9390923404a9e115eb as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f3/ec576c7193a240ca93fe66b5b7ba3739_SeqId_4_ 2016-10-20 14:34:22,386 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=37560] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974056208/f/1964b9cd70b2498ba1db9d3504b4624b as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f/8d28cd53ddb14f1aad8af155fbb5db3b_SeqId_4_ 2016-10-20 14:34:22,387 INFO [B.defaultRpcServer.handler=3,queue=0,port=37560] regionserver.HStore(742): Loaded HFile hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974056208/f3/0d18a694af6c4b9390923404a9e115eb into store 'f3' as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f3/ec576c7193a240ca93fe66b5b7ba3739_SeqId_4_ - updating store file list. 2016-10-20 14:34:22,387 INFO [B.defaultRpcServer.handler=4,queue=0,port=37560] regionserver.HStore(742): Loaded HFile hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974056208/f/1964b9cd70b2498ba1db9d3504b4624b into store 'f' as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f/8d28cd53ddb14f1aad8af155fbb5db3b_SeqId_4_ - updating store file list. 2016-10-20 14:34:22,392 INFO [B.defaultRpcServer.handler=3,queue=0,port=37560] regionserver.StoreFile(500): HFile Bloom filter type for ec576c7193a240ca93fe66b5b7ba3739_SeqId_4_: NONE, but ROW specified in column family configuration 2016-10-20 14:34:22,393 INFO [B.defaultRpcServer.handler=3,queue=0,port=37560] regionserver.HStore(777): Loaded HFile hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f3/ec576c7193a240ca93fe66b5b7ba3739_SeqId_4_ into store 'f3 2016-10-20 14:34:22,393 INFO [B.defaultRpcServer.handler=3,queue=0,port=37560] regionserver.HStore(748): Successfully loaded store file hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974056208/f3/0d18a694af6c4b9390923404a9e115eb into store f3 (new location: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f3/ec576c7193a240ca93fe66b5b7ba3739_SeqId_4_) 2016-10-20 14:34:22,393 INFO [B.defaultRpcServer.handler=4,queue=0,port=37560] regionserver.HStore(777): Loaded HFile hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f/8d28cd53ddb14f1aad8af155fbb5db3b_SeqId_4_ into store 'f 2016-10-20 14:34:22,393 INFO [B.defaultRpcServer.handler=4,queue=0,port=37560] regionserver.HStore(748): Successfully loaded store file hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974056208/f/1964b9cd70b2498ba1db9d3504b4624b into store f (new location: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f/8d28cd53ddb14f1aad8af155fbb5db3b_SeqId_4_) 2016-10-20 14:34:22,395 INFO [main] client.ConnectionImplementation(1805): Closing master protocol: MasterService 2016-10-20 14:34:22,395 INFO [main] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec60015 2016-10-20 14:34:22,402 DEBUG [main] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:22,404 DEBUG [main] mapreduce.MapReduceRestoreTask(106): Restore Job finished:0 2016-10-20 14:34:22,405 DEBUG [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42976 because read count=-1. Number of active connections: 9 2016-10-20 14:34:22,405 DEBUG [AsyncRpcChannel-pool2-t30] ipc.AsyncRpcChannel$8(566): IPC Client (1898134396) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:22,405 DEBUG [AsyncRpcChannel-pool2-t29] ipc.AsyncRpcChannel$8(566): IPC Client (-1226291460) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:22,405 DEBUG [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42972 because read count=-1. Number of active connections: 9 2016-10-20 14:34:22,405 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Listener(912): RpcServer.listener,port=37560: DISCONNECTING client 172.18.128.12:42348 because read count=-1. Number of active connections: 7 2016-10-20 14:34:22,405 DEBUG [AsyncRpcChannel-pool2-t31] ipc.AsyncRpcChannel$8(566): IPC Client (359501466) to cn012.l42scl.hortonworks.com/172.18.128.12:37560 from hbase: closed 2016-10-20 14:34:22,406 INFO [main] impl.RestoreTablesClient(168): Restoring 'ns1:test-1476974003890' to 'ns1:table1_restore' from log dirs: hdfs://localhost:37879/backupUT/backup_1476974042777/WALs 2016-10-20 14:34:22,412 DEBUG [main] util.RestoreServerUtil(219): Found descriptor 'ns1:test-1476974003890', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} through backup_1476974042777 2016-10-20 14:34:22,416 INFO [B.defaultRpcServer.handler=1,queue=0,port=41289] master.HMaster(2064): Client=hbase//172.18.128.12 modify ns1:table1_restore 2016-10-20 14:34:22,574 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure ModifyTableProcedure (table=ns1:table1_restore) id=19 owner=hbase state=RUNNABLE:MODIFY_TABLE_PREPARE added to the store. 2016-10-20 14:34:22,588 DEBUG [ProcedureExecutor-18] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/ns1:table1_restore/write-master:412890000000003 2016-10-20 14:34:22,909 INFO [IPC Server handler 0 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741920_1096{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:34:22,913 DEBUG [ProcedureExecutor-18] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/.tabledesc/.tableinfo.0000000002 2016-10-20 14:34:22,915 INFO [IPC Server handler 5 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741912_1088 127.0.0.1:38642 2016-10-20 14:34:22,916 DEBUG [ProcedureExecutor-18] util.FSTableDescriptors(664): Deleted table descriptor at hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/.tabledesc/.tableinfo.0000000001 2016-10-20 14:34:22,916 INFO [ProcedureExecutor-18] util.FSTableDescriptors(629): Updated tableinfo=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/.tabledesc/.tableinfo.0000000002 2016-10-20 14:34:23,203 DEBUG [ProcedureExecutor-18] procedure.MasterDDLOperationHelper(98): Removing family=f3 from table=ns1:table1_restore 2016-10-20 14:34:23,212 DEBUG [ProcedureExecutor-18] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f3/ec576c7193a240ca93fe66b5b7ba3739_SeqId_4_, to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/archive/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f3/ec576c7193a240ca93fe66b5b7ba3739_SeqId_4_ 2016-10-20 14:34:23,215 DEBUG [ProcedureExecutor-18] backup.HFileArchiver(167): No store files to dispose for region=ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066., family=f3 2016-10-20 14:34:23,525 INFO [ProcedureExecutor-18] procedure.MasterDDLOperationHelper(117): Bucketing regions by region server... 2016-10-20 14:34:23,530 INFO [ProcedureExecutor-18] procedure.MasterDDLOperationHelper(146): Reopening 2 regions on 1 region servers. 2016-10-20 14:34:23,530 DEBUG [ProcedureExecutor-18] master.BulkReOpen(57): Creating threads for each region server 2016-10-20 14:34:23,531 INFO [ProcedureExecutor-18] procedure.ModifyTableProcedure(437): Completed modify table operation on table ns1:table1_restore 2016-10-20 14:34:23,537 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.AssignmentManager(1352): Starting unassign of ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. (offlining), current state: {b35704e7c3c2eb86497b08776362fa99 state=OPEN, ts=1476974055807, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:23,537 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.RegionStates(1106): Transition {b35704e7c3c2eb86497b08776362fa99 state=OPEN, ts=1476974055807, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {b35704e7c3c2eb86497b08776362fa99 state=PENDING_CLOSE, ts=1476974063537, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:23,538 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. with state=PENDING_CLOSE 2016-10-20 14:34:23,541 INFO [PriorityRpcServer.handler=3,queue=1,port=37560] regionserver.RSRpcServices(1314): Close b35704e7c3c2eb86497b08776362fa99, moving to null 2016-10-20 14:34:23,541 DEBUG [RS_CLOSE_REGION-cn012:37560-1] handler.CloseRegionHandler(90): Processing close of ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:23,542 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.AssignmentManager(930): Sent CLOSE to cn012.l42scl.hortonworks.com,37560,1476973974890 for region ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:23,542 DEBUG [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(1419): Closing ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.: disabling compactions & flushes 2016-10-20 14:34:23,542 DEBUG [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(1446): Updates disabled for region ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:23,543 INFO [StoreCloserThread-ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.-1] regionserver.HStore(839): Closed f 2016-10-20 14:34:23,548 INFO [StoreCloserThread-ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.-1] regionserver.HStore(839): Closed f3 2016-10-20 14:34:23,558 DEBUG [RS_CLOSE_REGION-cn012:37560-1] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/recovered.edits/6.seqid to file, newSeqId=6, maxSeqId=2 2016-10-20 14:34:23,560 INFO [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(1552): Closed ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:23,561 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.AssignmentManager(2884): Got transition CLOSED for {b35704e7c3c2eb86497b08776362fa99 state=PENDING_CLOSE, ts=1476974063537, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:23,562 INFO [B.defaultRpcServer.handler=3,queue=0,port=41289] master.RegionStates(1106): Transition {b35704e7c3c2eb86497b08776362fa99 state=PENDING_CLOSE, ts=1476974063537, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {b35704e7c3c2eb86497b08776362fa99 state=CLOSED, ts=1476974063562, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:23,563 INFO [B.defaultRpcServer.handler=3,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. with state=CLOSED 2016-10-20 14:34:23,565 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=41289] master.AssignmentManager(603): Removed region from reopening regions because it was closed 2016-10-20 14:34:23,566 DEBUG [RS_CLOSE_REGION-cn012:37560-1] handler.CloseRegionHandler(122): Closed ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:23,569 DEBUG [AM.-pool3-t3] master.AssignmentManager(1260): Found an existing plan for ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. destination server is cn012.l42scl.hortonworks.com,37560,1476973974890 accepted as a dest server = true 2016-10-20 14:34:23,569 DEBUG [AM.-pool3-t3] master.AssignmentManager(1300): Using pre-existing plan for ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.; plan=hri=ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99., src=, dest=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:23,569 INFO [AM.-pool3-t3] master.AssignmentManager(1080): Assigning ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. to cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:23,569 INFO [AM.-pool3-t3] master.RegionStates(1106): Transition {b35704e7c3c2eb86497b08776362fa99 state=CLOSED, ts=1476974063562, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {b35704e7c3c2eb86497b08776362fa99 state=PENDING_OPEN, ts=1476974063569, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:23,569 INFO [AM.-pool3-t3] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. with state=PENDING_OPEN 2016-10-20 14:34:23,572 INFO [PriorityRpcServer.handler=1,queue=1,port=37560] regionserver.RSRpcServices(1666): Open ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:23,581 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(6339): Opening region: {ENCODED => b35704e7c3c2eb86497b08776362fa99, NAME => 'ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.', STARTKEY => '', ENDKEY => 'row0'} 2016-10-20 14:34:23,581 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table table1_restore b35704e7c3c2eb86497b08776362fa99 2016-10-20 14:34:23,582 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(736): Instantiated ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:23,585 INFO [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:23,585 INFO [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:23,587 DEBUG [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f 2016-10-20 14:34:23,590 INFO [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:23,590 INFO [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:23,591 DEBUG [StoreOpener-b35704e7c3c2eb86497b08776362fa99-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f2 2016-10-20 14:34:23,594 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99 2016-10-20 14:34:23,597 DEBUG [RS_OPEN_REGION-cn012:37560-0] regionserver.FlushLargeStoresPolicy(72): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table ns1:table1_restore, use config (67108864) instead 2016-10-20 14:34:23,603 DEBUG [RS_OPEN_REGION-cn012:37560-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/recovered.edits/7.seqid to file, newSeqId=7, maxSeqId=6 2016-10-20 14:34:23,605 INFO [RS_OPEN_REGION-cn012:37560-0] regionserver.HRegion(871): Onlined b35704e7c3c2eb86497b08776362fa99; next sequenceid=7 2016-10-20 14:34:23,607 INFO [PostOpenDeployTasks:b35704e7c3c2eb86497b08776362fa99] regionserver.HRegionServer(1957): Post open deploy tasks for ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:23,608 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.AssignmentManager(2884): Got transition OPENED for {b35704e7c3c2eb86497b08776362fa99 state=PENDING_OPEN, ts=1476974063569, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:23,608 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.RegionStates(1106): Transition {b35704e7c3c2eb86497b08776362fa99 state=PENDING_OPEN, ts=1476974063569, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {b35704e7c3c2eb86497b08776362fa99 state=OPEN, ts=1476974063608, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:23,609 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. with state=OPEN, openSeqNum=7, server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:23,624 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.AssignmentManager(1352): Starting unassign of ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. (offlining), current state: {a1135d09b318d77673368ed569f0b066 state=OPEN, ts=1476974055809, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:23,624 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.RegionStates(1106): Transition {a1135d09b318d77673368ed569f0b066 state=OPEN, ts=1476974055809, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {a1135d09b318d77673368ed569f0b066 state=PENDING_CLOSE, ts=1476974063624, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:23,624 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. with state=PENDING_CLOSE 2016-10-20 14:34:23,625 DEBUG [PostOpenDeployTasks:b35704e7c3c2eb86497b08776362fa99] regionserver.HRegionServer(1984): Finished post open deploy task for ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:23,628 DEBUG [RS_OPEN_REGION-cn012:37560-0] handler.OpenRegionHandler(126): Opened ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:23,629 INFO [PriorityRpcServer.handler=2,queue=0,port=37560] regionserver.RSRpcServices(1314): Close a1135d09b318d77673368ed569f0b066, moving to null 2016-10-20 14:34:23,630 DEBUG [RS_CLOSE_REGION-cn012:37560-0] handler.CloseRegionHandler(90): Processing close of ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:23,630 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.BulkReOpen-0] master.AssignmentManager(930): Sent CLOSE to cn012.l42scl.hortonworks.com,37560,1476973974890 for region ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:23,630 DEBUG [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(1419): Closing ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.: disabling compactions & flushes 2016-10-20 14:34:23,630 DEBUG [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(1446): Updates disabled for region ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:23,636 INFO [StoreCloserThread-ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.-1] regionserver.HStore(839): Closed f 2016-10-20 14:34:23,636 INFO [StoreCloserThread-ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.-1] regionserver.HStore(839): Closed f3 2016-10-20 14:34:23,647 DEBUG [RS_CLOSE_REGION-cn012:37560-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/recovered.edits/6.seqid to file, newSeqId=6, maxSeqId=2 2016-10-20 14:34:23,649 INFO [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(1552): Closed ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:23,650 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.AssignmentManager(2884): Got transition CLOSED for {a1135d09b318d77673368ed569f0b066 state=PENDING_CLOSE, ts=1476974063624, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:23,651 INFO [B.defaultRpcServer.handler=4,queue=0,port=41289] master.RegionStates(1106): Transition {a1135d09b318d77673368ed569f0b066 state=PENDING_CLOSE, ts=1476974063624, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {a1135d09b318d77673368ed569f0b066 state=CLOSED, ts=1476974063651, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:23,651 INFO [B.defaultRpcServer.handler=4,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. with state=CLOSED 2016-10-20 14:34:23,652 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.AssignmentManager(603): Removed region from reopening regions because it was closed 2016-10-20 14:34:23,653 DEBUG [RS_CLOSE_REGION-cn012:37560-0] handler.CloseRegionHandler(122): Closed ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:23,656 DEBUG [AM.-pool3-t4] master.AssignmentManager(1260): Found an existing plan for ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. destination server is cn012.l42scl.hortonworks.com,37560,1476973974890 accepted as a dest server = true 2016-10-20 14:34:23,656 DEBUG [AM.-pool3-t4] master.AssignmentManager(1300): Using pre-existing plan for ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.; plan=hri=ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066., src=, dest=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:23,657 INFO [AM.-pool3-t4] master.AssignmentManager(1080): Assigning ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. to cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:23,657 INFO [AM.-pool3-t4] master.RegionStates(1106): Transition {a1135d09b318d77673368ed569f0b066 state=CLOSED, ts=1476974063651, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {a1135d09b318d77673368ed569f0b066 state=PENDING_OPEN, ts=1476974063657, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:23,657 INFO [AM.-pool3-t4] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. with state=PENDING_OPEN 2016-10-20 14:34:23,659 INFO [PriorityRpcServer.handler=3,queue=1,port=37560] regionserver.RSRpcServices(1666): Open ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:23,668 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(6339): Opening region: {ENCODED => a1135d09b318d77673368ed569f0b066, NAME => 'ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.', STARTKEY => 'row0', ENDKEY => ''} 2016-10-20 14:34:23,668 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table table1_restore a1135d09b318d77673368ed569f0b066 2016-10-20 14:34:23,669 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(736): Instantiated ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:23,672 INFO [StoreOpener-a1135d09b318d77673368ed569f0b066-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:23,673 INFO [StoreOpener-a1135d09b318d77673368ed569f0b066-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:23,686 DEBUG [StoreOpener-a1135d09b318d77673368ed569f0b066-1] regionserver.HStore(499): loaded hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f/8d28cd53ddb14f1aad8af155fbb5db3b_SeqId_4_, isReference=false, isBulkLoadResult=true, bulkLoadTS=1476974061790, majorCompaction=true 2016-10-20 14:34:23,688 INFO [StoreOpener-a1135d09b318d77673368ed569f0b066-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:23,689 INFO [StoreOpener-a1135d09b318d77673368ed569f0b066-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:23,690 DEBUG [StoreOpener-a1135d09b318d77673368ed569f0b066-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/f2 2016-10-20 14:34:23,693 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066 2016-10-20 14:34:23,694 DEBUG [ProcedureExecutor-18] lock.ZKInterProcessLockBase(328): Released /1/table-lock/ns1:table1_restore/write-master:412890000000003 2016-10-20 14:34:23,695 DEBUG [ProcedureExecutor-18] procedure2.ProcedureExecutor(870): Procedure completed in 1.1210sec: ModifyTableProcedure (table=ns1:table1_restore) id=19 owner=hbase state=FINISHED 2016-10-20 14:34:23,696 DEBUG [RS_OPEN_REGION-cn012:37560-1] regionserver.FlushLargeStoresPolicy(72): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of table ns1:table1_restore, use config (67108864) instead 2016-10-20 14:34:23,702 DEBUG [RS_OPEN_REGION-cn012:37560-1] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/recovered.edits/7.seqid to file, newSeqId=7, maxSeqId=6 2016-10-20 14:34:23,703 INFO [RS_OPEN_REGION-cn012:37560-1] regionserver.HRegion(871): Onlined a1135d09b318d77673368ed569f0b066; next sequenceid=7 2016-10-20 14:34:23,705 INFO [PostOpenDeployTasks:a1135d09b318d77673368ed569f0b066] regionserver.HRegionServer(1957): Post open deploy tasks for ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:23,706 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.AssignmentManager(2884): Got transition OPENED for {a1135d09b318d77673368ed569f0b066 state=PENDING_OPEN, ts=1476974063657, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:23,707 INFO [B.defaultRpcServer.handler=2,queue=0,port=41289] master.RegionStates(1106): Transition {a1135d09b318d77673368ed569f0b066 state=PENDING_OPEN, ts=1476974063657, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {a1135d09b318d77673368ed569f0b066 state=OPEN, ts=1476974063707, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:23,707 INFO [B.defaultRpcServer.handler=2,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. with state=OPEN, openSeqNum=7, server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:23,710 DEBUG [PostOpenDeployTasks:a1135d09b318d77673368ed569f0b066] regionserver.HRegionServer(1984): Finished post open deploy task for ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:23,713 DEBUG [RS_OPEN_REGION-cn012:37560-1] handler.OpenRegionHandler(126): Opened ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:23,835 DEBUG [main] client.ConnectionImplementation(604): Table ns1:table1_restore should be available 2016-10-20 14:34:23,836 INFO [main] util.RestoreServerUtil(241): Changed ns1:table1_restore to: 'ns1:table1_restore', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2016-10-20 14:34:23,841 DEBUG [main] mapreduce.MapReduceRestoreTask(68): Restore incremental backup from directory hdfs://localhost:37879/backupUT/backup_1476974042777/WALs from hbase tables ,ns1:test-1476974003890 to tables ,ns1:table1_restore 2016-10-20 14:34:23,841 INFO [main] mapreduce.MapReduceRestoreTask(76): Restore ns1:test-1476974003890 into ns1:table1_restore 2016-10-20 14:34:23,845 DEBUG [main] mapreduce.WALPlayer(299): add incremental job :/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974063841 from hdfs://localhost:37879/backupUT/backup_1476974042777/WALs 2016-10-20 14:34:23,847 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x45843650 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:34:23,861 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x458436500x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:34:23,865 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x45843650-0x157e2823ec60016 connected 2016-10-20 14:34:23,865 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72443081, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:34:23,866 DEBUG [main] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:34:23,866 DEBUG [main] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:34:23,871 DEBUG [AsyncRpcChannel-pool2-t32] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service MasterService, sasl=false 2016-10-20 14:34:23,871 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42990; # active connections: 8 2016-10-20 14:34:23,874 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:23,875 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42990 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:23,878 INFO [main] mapreduce.HFileOutputFormat2(478): bulkload locality sensitive enabled 2016-10-20 14:34:23,878 INFO [main] mapreduce.HFileOutputFormat2(483): Looking up current regions for table ns1:test-1476974003890 2016-10-20 14:34:23,892 DEBUG [AsyncRpcChannel-pool2-t33] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:23,892 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:42992; # active connections: 9 2016-10-20 14:34:23,895 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:23,895 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42992 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:23,902 INFO [main] mapreduce.HFileOutputFormat2(485): Configuring 1 reduce partitions to match current region count 2016-10-20 14:34:23,903 INFO [main] mapreduce.HFileOutputFormat2(378): Writing partition information to /user/hbase/hbase-staging/partitions_2bf84ef9-6181-4705-82ab-508803ec37ba 2016-10-20 14:34:23,914 INFO [IPC Server handler 9 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741921_1097{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 153 2016-10-20 14:34:24,318 WARN [main] mapreduce.TableMapReduceUtil(786): The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it. 2016-10-20 14:34:24,509 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.HConstants, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-8531529338740817810.jar 2016-10-20 14:34:24,614 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@3dddefd8] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:38642 to delete [blk_1073741912_1088] 2016-10-20 14:34:25,369 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-132084151543827469.jar 2016-10-20 14:34:25,743 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.client.Put, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-5359034022477017526.jar 2016-10-20 14:34:25,760 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.CompatibilityFactory, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-2237925754273609492.jar 2016-10-20 14:34:26,764 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-6740843940742236200.jar 2016-10-20 14:34:26,765 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.zookeeper.ZooKeeper, using jar /home/hbase/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar 2016-10-20 14:34:26,765 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class io.netty.channel.Channel, using jar /home/hbase/.m2/repository/io/netty/netty-all/4.0.30.Final/netty-all-4.0.30.Final.jar 2016-10-20 14:34:26,766 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.protobuf.Message, using jar /home/hbase/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar 2016-10-20 14:34:26,766 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.common.collect.Lists, using jar /home/hbase/.m2/repository/com/google/guava/guava/12.0.1/guava-12.0.1.jar 2016-10-20 14:34:26,766 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.htrace.Trace, using jar /home/hbase/.m2/repository/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar 2016-10-20 14:34:26,767 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.codahale.metrics.MetricRegistry, using jar /home/hbase/.m2/repository/io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.jar 2016-10-20 14:34:26,904 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-5878029848411028959.jar 2016-10-20 14:34:26,905 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.KeyValue, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-5878029848411028959.jar 2016-10-20 14:34:27,795 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.mapreduce.WALInputFormat, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-8738451378368200648.jar 2016-10-20 14:34:27,796 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-5878029848411028959.jar 2016-10-20 14:34:27,797 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.KeyValue, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-5878029848411028959.jar 2016-10-20 14:34:27,797 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-8738451378368200648.jar 2016-10-20 14:34:27,798 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.1/hadoop-mapreduce-client-core-2.7.1.jar 2016-10-20 14:34:27,798 INFO [main] mapreduce.HFileOutputFormat2(498): Incremental table ns1:test-1476974003890 output configured. 2016-10-20 14:34:27,798 INFO [main] client.ConnectionImplementation(1805): Closing master protocol: MasterService 2016-10-20 14:34:27,799 INFO [main] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec60016 2016-10-20 14:34:27,819 DEBUG [main] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:27,823 DEBUG [main] mapreduce.WALPlayer(316): success configuring load incremental job 2016-10-20 14:34:27,823 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42990 because read count=-1. Number of active connections: 9 2016-10-20 14:34:27,824 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.common.base.Preconditions, using jar /home/hbase/.m2/repository/com/google/guava/guava/12.0.1/guava-12.0.1.jar 2016-10-20 14:34:27,823 DEBUG [AsyncRpcChannel-pool2-t33] ipc.AsyncRpcChannel$8(566): IPC Client (715404387) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:27,823 DEBUG [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42992 because read count=-1. Number of active connections: 9 2016-10-20 14:34:27,823 DEBUG [AsyncRpcChannel-pool2-t32] ipc.AsyncRpcChannel$8(566): IPC Client (-703383675) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:27,838 WARN [main] mapreduce.JobResourceUploader(64): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-10-20 14:34:27,854 WARN [main] mapreduce.JobResourceUploader(171): No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-10-20 14:34:28,251 DEBUG [main] mapreduce.WALInputFormat(265): Scanning hdfs://localhost:37879/backupUT/backup_1476974042777/WALs for WAL files 2016-10-20 14:34:28,255 WARN [main] mapreduce.WALInputFormat(289): File hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/.backup.manifest does not appear to be an WAL file. Skipping... 2016-10-20 14:34:28,256 INFO [main] mapreduce.WALInputFormat(281): Found: LocatedFileStatus{path=hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974036606; isDirectory=false; length=1734; replication=1; blocksize=134217728; modification_time=1476974045719; access_time=1476974045304; owner=hbase; group=supergroup; permission=rw-r--r--; isSymlink=false} 2016-10-20 14:34:28,256 INFO [main] mapreduce.WALInputFormat(281): Found: LocatedFileStatus{path=hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974037083; isDirectory=false; length=5239; replication=1; blocksize=134217728; modification_time=1476974045753; access_time=1476974045741; owner=hbase; group=supergroup; permission=rw-r--r--; isSymlink=false} 2016-10-20 14:34:28,256 INFO [main] mapreduce.WALInputFormat(281): Found: LocatedFileStatus{path=hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974036606; isDirectory=false; length=91; replication=1; blocksize=134217728; modification_time=1476974045782; access_time=1476974045771; owner=hbase; group=supergroup; permission=rw-r--r--; isSymlink=false} 2016-10-20 14:34:28,256 INFO [main] mapreduce.WALInputFormat(281): Found: LocatedFileStatus{path=hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974036670; isDirectory=false; length=91; replication=1; blocksize=134217728; modification_time=1476974045812; access_time=1476974045801; owner=hbase; group=supergroup; permission=rw-r--r--; isSymlink=false} 2016-10-20 14:34:29,065 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-jobhistoryserver.properties,hadoop-metrics2.properties 2016-10-20 14:34:29,225 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(152): Opening reader for hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974037083 (-9223372036854775808:9223372036854775807) length:5239 2016-10-20 14:34:29,234 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(185): Reached end of file. 2016-10-20 14:34:29,235 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(210): Closing reader 2016-10-20 14:34:29,257 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(152): Opening reader for hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974036606 (-9223372036854775808:9223372036854775807) length:1734 2016-10-20 14:34:29,263 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(185): Reached end of file. 2016-10-20 14:34:29,264 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(210): Closing reader 2016-10-20 14:34:29,420 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(152): Opening reader for hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974036606 (-9223372036854775808:9223372036854775807) length:91 2016-10-20 14:34:29,438 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(185): Reached end of file. 2016-10-20 14:34:29,439 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(210): Closing reader 2016-10-20 14:34:29,460 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(152): Opening reader for hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974036670 (-9223372036854775808:9223372036854775807) length:91 2016-10-20 14:34:29,463 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(185): Reached end of file. 2016-10-20 14:34:29,464 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(210): Closing reader 2016-10-20 14:34:29,499 INFO [pool-195-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x3bec4367 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:34:29,945 DEBUG [pool-195-thread-1-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x3bec43670x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:34:29,949 DEBUG [pool-195-thread-1-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x3bec4367-0x157e2823ec60017 connected 2016-10-20 14:34:29,949 DEBUG [pool-195-thread-1] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@447b3eb6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:34:29,949 DEBUG [pool-195-thread-1] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:34:29,950 DEBUG [pool-195-thread-1] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:34:29,958 DEBUG [AsyncRpcChannel-pool2-t34] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:29,958 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:43014; # active connections: 8 2016-10-20 14:34:29,961 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:29,961 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 43014 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:29,965 INFO [pool-195-thread-1] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec60017 2016-10-20 14:34:29,969 DEBUG [pool-195-thread-1] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:29,974 DEBUG [pool-195-thread-1] mapreduce.HFileOutputFormat2$1(232): first rowkey: [row-ns1:test-1476974003890-2-0] 2016-10-20 14:34:29,974 DEBUG [pool-195-thread-1] mapreduce.HFileOutputFormat2$1(244): use favored nodes writer: cn012.l42scl.hortonworks.com 2016-10-20 14:34:29,974 DEBUG [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:43014 because read count=-1. Number of active connections: 8 2016-10-20 14:34:29,974 DEBUG [AsyncRpcChannel-pool2-t34] ipc.AsyncRpcChannel$8(566): IPC Client (-1156386788) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:29,975 INFO [pool-195-thread-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:29,986 WARN [Thread-2288] hdfs.DFSOutputStream$DataStreamer(1413): These favored nodes were specified but not chosen: [cn012.l42scl.hortonworks.com:37560] Specified favored nodes: [cn012.l42scl.hortonworks.com:37560] 2016-10-20 14:34:29,991 INFO [IPC Server handler 1 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741922_1098{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:34:30,269 DEBUG [main] mapreduce.MapReduceRestoreTask(93): Restoring HFiles from directory /user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974063841 2016-10-20 14:34:30,269 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x20801cbb connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:34:30,286 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x20801cbb0x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:34:30,290 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x20801cbb-0x157e2823ec60018 connected 2016-10-20 14:34:30,290 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@581b1c08, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:34:30,290 DEBUG [main] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:34:30,290 DEBUG [main] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:34:30,296 DEBUG [AsyncRpcChannel-pool2-t35] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:30,296 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:43020; # active connections: 8 2016-10-20 14:34:30,299 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:30,300 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 43020 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:30,316 DEBUG [main] client.ConnectionImplementation(604): Table ns1:table1_restore should be available 2016-10-20 14:34:30,319 WARN [main] mapreduce.LoadIncrementalHFiles(202): Skipping non-directory hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974063841/_SUCCESS 2016-10-20 14:34:30,331 DEBUG [AsyncRpcChannel-pool2-t36] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service MasterService, sasl=false 2016-10-20 14:34:30,331 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:43022; # active connections: 9 2016-10-20 14:34:30,334 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:30,335 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 43022 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:30,344 INFO [LoadIncrementalHFiles-0] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:30,351 INFO [LoadIncrementalHFiles-0] mapreduce.LoadIncrementalHFiles(820): Trying to load hfile=hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974063841/f2/63f4378eb301465ca9bb329dad393300 first=row-ns1:test-1476974003890-2-0 last=row-ns1:test-1476974003890-2-6 2016-10-20 14:34:30,361 DEBUG [LoadIncrementalHFiles-1] mapreduce.LoadIncrementalHFiles$4(913): Going to connect to server region=ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99., hostname=cn012.l42scl.hortonworks.com,37560,1476973974890, seqNum=7 for row with hfile group [{[B@3caafa67,hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974063841/f2/63f4378eb301465ca9bb329dad393300}] 2016-10-20 14:34:30,368 DEBUG [AsyncRpcChannel-pool2-t37] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:30,369 DEBUG [RpcServer.listener,port=37560] ipc.RpcServer$Listener(880): RpcServer.listener,port=37560: connection from 172.18.128.12:42392; # active connections: 7 2016-10-20 14:34:30,372 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:30,372 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42392 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:30,373 INFO [B.defaultRpcServer.handler=0,queue=0,port=37560] regionserver.HStore(670): Validating hfile at hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974063841/f2/63f4378eb301465ca9bb329dad393300 for inclusion in store f2 region ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:30,378 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=37560] regionserver.HStore(682): HFile bounds: first=row-ns1:test-1476974003890-2-0 last=row-ns1:test-1476974003890-2-6 2016-10-20 14:34:30,379 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=37560] regionserver.HStore(684): Region bounds: first= last=row0 2016-10-20 14:34:30,383 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=37560] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974063841/f2/63f4378eb301465ca9bb329dad393300 as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f2/ebab33d6e74b4a869ccbbadab583d78c_SeqId_9_ 2016-10-20 14:34:30,385 INFO [B.defaultRpcServer.handler=0,queue=0,port=37560] regionserver.HStore(742): Loaded HFile hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974063841/f2/63f4378eb301465ca9bb329dad393300 into store 'f2' as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f2/ebab33d6e74b4a869ccbbadab583d78c_SeqId_9_ - updating store file list. 2016-10-20 14:34:30,395 INFO [B.defaultRpcServer.handler=0,queue=0,port=37560] regionserver.HStore(777): Loaded HFile hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f2/ebab33d6e74b4a869ccbbadab583d78c_SeqId_9_ into store 'f2 2016-10-20 14:34:30,395 INFO [B.defaultRpcServer.handler=0,queue=0,port=37560] regionserver.HStore(748): Successfully loaded store file hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns1-table1_restore-1476974063841/f2/63f4378eb301465ca9bb329dad393300 into store f2 (new location: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/f2/ebab33d6e74b4a869ccbbadab583d78c_SeqId_9_) 2016-10-20 14:34:30,398 INFO [main] client.ConnectionImplementation(1805): Closing master protocol: MasterService 2016-10-20 14:34:30,399 INFO [main] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec60018 2016-10-20 14:34:30,402 DEBUG [main] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:30,405 DEBUG [main] mapreduce.MapReduceRestoreTask(106): Restore Job finished:0 2016-10-20 14:34:30,406 INFO [main] impl.RestoreTablesClient(173): ns1:test-1476974003890 has been successfully restored to ns1:table1_restore 2016-10-20 14:34:30,406 DEBUG [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:43022 because read count=-1. Number of active connections: 9 2016-10-20 14:34:30,406 DEBUG [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Listener(912): RpcServer.listener,port=37560: DISCONNECTING client 172.18.128.12:42392 because read count=-1. Number of active connections: 7 2016-10-20 14:34:30,406 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:43020 because read count=-1. Number of active connections: 9 2016-10-20 14:34:30,406 DEBUG [AsyncRpcChannel-pool2-t36] ipc.AsyncRpcChannel$8(566): IPC Client (-1518757026) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:30,406 DEBUG [AsyncRpcChannel-pool2-t35] ipc.AsyncRpcChannel$8(566): IPC Client (-972581691) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:30,406 INFO [main] impl.RestoreTablesClient(205): Restore includes the following image(s): 2016-10-20 14:34:30,407 INFO [main] impl.RestoreTablesClient(207): Backup: backup_1476974021410 hdfs://localhost:37879/backupUT/backup_1476974021410/ns1/test-1476974003890/ 2016-10-20 14:34:30,406 DEBUG [AsyncRpcChannel-pool2-t37] ipc.AsyncRpcChannel$8(566): IPC Client (-1080317748) to cn012.l42scl.hortonworks.com/172.18.128.12:37560 from hbase: closed 2016-10-20 14:34:30,408 DEBUG [main] impl.BackupManifest(348): Loading manifest from: hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901/.backup.manifest 2016-10-20 14:34:30,413 DEBUG [main] impl.BackupManifest(402): Loaded manifest instance from manifest file: /backupUT/backup_1476974021410/ns2/test-14769740038901/.backup.manifest 2016-10-20 14:34:30,413 INFO [main] impl.RestoreTablesClient(145): Restoring 'ns2:test-14769740038901' to 'ns2:table2_restore' from full backup image hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901 2016-10-20 14:34:30,417 DEBUG [main] util.RestoreServerUtil(383): Retrieved descriptor: 'ns2:test-14769740038901', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} thru backup_1476974042777 2016-10-20 14:34:30,422 INFO [main] util.RestoreServerUtil(696): Truncating exising target table 'ns2:table2_restore', preserving region splits 2016-10-20 14:34:30,423 INFO [main] client.HBaseAdmin$10(773): Started disable of ns2:table2_restore 2016-10-20 14:34:30,424 INFO [B.defaultRpcServer.handler=0,queue=0,port=41289] master.HMaster(1992): Client=hbase//172.18.128.12 disable ns2:table2_restore 2016-10-20 14:34:30,591 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure DisableTableProcedure (table=ns2:table2_restore) id=20 owner=hbase state=RUNNABLE:DISABLE_TABLE_PREPARE added to the store. 2016-10-20 14:34:30,597 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=20 2016-10-20 14:34:30,613 DEBUG [ProcedureExecutor-19] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/ns2:table2_restore/write-master:412890000000001 2016-10-20 14:34:30,699 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=20 2016-10-20 14:34:30,901 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=20 2016-10-20 14:34:30,916 DEBUG [ProcedureExecutor-19] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"ns2:table2_restore","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476974070916}]}} 2016-10-20 14:34:30,920 INFO [ProcedureExecutor-19] hbase.MetaTableAccessor(1700): Updated table ns2:table2_restore state to DISABLING in META 2016-10-20 14:34:31,066 INFO [ProcedureExecutor-19] procedure.DisableTableProcedure(395): Offlining 1 regions. 2016-10-20 14:34:31,070 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.AssignmentManager(1352): Starting unassign of ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. (offlining), current state: {81900e9262bb3c77f4b3fd7b6387aeb8 state=OPEN, ts=1476974051107, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:31,070 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.RegionStates(1106): Transition {81900e9262bb3c77f4b3fd7b6387aeb8 state=OPEN, ts=1476974051107, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {81900e9262bb3c77f4b3fd7b6387aeb8 state=PENDING_CLOSE, ts=1476974071070, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:31,071 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.RegionStateStore(207): Updating hbase:meta row ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. with state=PENDING_CLOSE 2016-10-20 14:34:31,074 INFO [PriorityRpcServer.handler=1,queue=1,port=37560] regionserver.RSRpcServices(1314): Close 81900e9262bb3c77f4b3fd7b6387aeb8, moving to null 2016-10-20 14:34:31,074 DEBUG [RS_CLOSE_REGION-cn012:37560-2] handler.CloseRegionHandler(90): Processing close of ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:31,074 DEBUG [cn012.l42scl.hortonworks.com,41289,1476973974467-org.apache.hadoop.hbase.master.procedure.DisableTableProcedure$BulkDisabler-0] master.AssignmentManager(930): Sent CLOSE to cn012.l42scl.hortonworks.com,37560,1476973974890 for region ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:31,075 DEBUG [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1419): Closing ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8.: disabling compactions & flushes 2016-10-20 14:34:31,075 DEBUG [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1446): Updates disabled for region ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:31,079 INFO [StoreCloserThread-ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8.-1] regionserver.HStore(839): Closed f 2016-10-20 14:34:31,090 DEBUG [RS_CLOSE_REGION-cn012:37560-2] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/recovered.edits/6.seqid to file, newSeqId=6, maxSeqId=2 2016-10-20 14:34:31,092 INFO [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1552): Closed ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:31,093 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.AssignmentManager(2884): Got transition CLOSED for {81900e9262bb3c77f4b3fd7b6387aeb8 state=PENDING_CLOSE, ts=1476974071070, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:31,095 INFO [B.defaultRpcServer.handler=1,queue=0,port=41289] master.RegionStates(1106): Transition {81900e9262bb3c77f4b3fd7b6387aeb8 state=PENDING_CLOSE, ts=1476974071070, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {81900e9262bb3c77f4b3fd7b6387aeb8 state=OFFLINE, ts=1476974071095, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:31,095 INFO [B.defaultRpcServer.handler=1,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. with state=OFFLINE 2016-10-20 14:34:31,097 INFO [B.defaultRpcServer.handler=1,queue=0,port=41289] master.RegionStates(590): Offlined 81900e9262bb3c77f4b3fd7b6387aeb8 from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:31,098 DEBUG [RS_CLOSE_REGION-cn012:37560-2] handler.CloseRegionHandler(122): Closed ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:31,203 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=20 2016-10-20 14:34:31,258 DEBUG [ProcedureExecutor-19] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"ns2:table2_restore","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476974071257}]}} 2016-10-20 14:34:31,260 INFO [ProcedureExecutor-19] hbase.MetaTableAccessor(1700): Updated table ns2:table2_restore state to DISABLED in META 2016-10-20 14:34:31,261 INFO [ProcedureExecutor-19] procedure.DisableTableProcedure(424): Disabled table, ns2:table2_restore, is completed. 2016-10-20 14:34:31,586 DEBUG [ProcedureExecutor-19] lock.ZKInterProcessLockBase(328): Released /1/table-lock/ns2:table2_restore/write-master:412890000000001 2016-10-20 14:34:31,586 DEBUG [ProcedureExecutor-19] procedure2.ProcedureExecutor(870): Procedure completed in 1.0040sec: DisableTableProcedure (table=ns2:table2_restore) id=20 owner=hbase state=FINISHED 2016-10-20 14:34:31,706 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=20 2016-10-20 14:34:31,707 INFO [main] client.HBaseAdmin$TableFuture(3337): Operation: DISABLE, Table Name: ns2:table2_restore completed 2016-10-20 14:34:31,708 INFO [main] client.HBaseAdmin$8(608): Started truncating ns2:table2_restore 2016-10-20 14:34:31,709 INFO [B.defaultRpcServer.handler=4,queue=0,port=41289] master.HMaster(1854): Client=hbase//172.18.128.12 truncate ns2:table2_restore 2016-10-20 14:34:31,861 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=41289] procedure2.ProcedureExecutor(669): Procedure TruncateTableProcedure (table=ns2:table2_restore preserveSplits=true) id=21 owner=hbase state=RUNNABLE:TRUNCATE_TABLE_PRE_OPERATION added to the store. 2016-10-20 14:34:31,871 DEBUG [ProcedureExecutor-20] lock.ZKInterProcessLockBase(226): Acquired a lock for /1/table-lock/ns2:table2_restore/write-master:412890000000002 2016-10-20 14:34:31,873 DEBUG [ProcedureExecutor-20] procedure.TruncateTableProcedure(87): waiting for 'ns2:table2_restore' regions in transition 2016-10-20 14:34:32,028 DEBUG [ProcedureExecutor-20] hbase.MetaTableAccessor(1406): Delete{"totalColumns":1,"row":"ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":1476974072027}]},"ts":9223372036854775807} 2016-10-20 14:34:32,031 INFO [ProcedureExecutor-20] hbase.MetaTableAccessor(1854): Deleted [{ENCODED => 81900e9262bb3c77f4b3fd7b6387aeb8, NAME => 'ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8.', STARTKEY => '', ENDKEY => ''}] 2016-10-20 14:34:32,033 DEBUG [ProcedureExecutor-20] procedure.DeleteTableProcedure(408): Removing 'ns2:table2_restore' from region states. 2016-10-20 14:34:32,037 DEBUG [ProcedureExecutor-20] procedure.DeleteTableProcedure(412): Marking 'ns2:table2_restore' as deleted. 2016-10-20 14:34:32,037 DEBUG [ProcedureExecutor-20] hbase.MetaTableAccessor(1406): Delete{"totalColumns":1,"row":"ns2:table2_restore","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":1476974072037}]},"ts":9223372036854775807} 2016-10-20 14:34:32,040 INFO [ProcedureExecutor-20] hbase.MetaTableAccessor(1726): Deleted table ns2:table2_restore state from META 2016-10-20 14:34:32,191 DEBUG [ProcedureExecutor-20] procedure.DeleteTableProcedure(340): Archiving region ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. from FS 2016-10-20 14:34:32,191 DEBUG [ProcedureExecutor-20] backup.HFileArchiver(93): ARCHIVING hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8 2016-10-20 14:34:32,196 DEBUG [ProcedureExecutor-20] backup.HFileArchiver(134): Archiving [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/f, class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/recovered.edits] 2016-10-20 14:34:32,209 DEBUG [ProcedureExecutor-20] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/f/f821ebe52bba4f08b45eba7f62321282_SeqId_4_, to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/archive/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/f/f821ebe52bba4f08b45eba7f62321282_SeqId_4_ 2016-10-20 14:34:32,218 DEBUG [ProcedureExecutor-20] backup.HFileArchiver(438): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/recovered.edits/6.seqid, to hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/archive/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/recovered.edits/6.seqid 2016-10-20 14:34:32,219 INFO [IPC Server handler 7 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741910_1086 127.0.0.1:38642 2016-10-20 14:34:32,220 DEBUG [ProcedureExecutor-20] backup.HFileArchiver(453): Deleted all region files in: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8 2016-10-20 14:34:32,220 DEBUG [ProcedureExecutor-20] procedure.DeleteTableProcedure(344): Table 'ns2:table2_restore' archived! 2016-10-20 14:34:32,222 INFO [IPC Server handler 9 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741909_1085 127.0.0.1:38642 2016-10-20 14:34:32,382 INFO [IPC Server handler 4 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741923_1099{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 0 2016-10-20 14:34:32,386 DEBUG [ProcedureExecutor-20] util.FSTableDescriptors(718): Wrote descriptor into: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp/data/ns2/table2_restore/.tabledesc/.tableinfo.0000000001 2016-10-20 14:34:32,388 INFO [RegionOpenAndInitThread-ns2:table2_restore-1] regionserver.HRegion(6162): creating HRegion ns2:table2_restore HTD == 'ns2:table2_restore', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/.tmp Table name == ns2:table2_restore 2016-10-20 14:34:32,404 INFO [IPC Server handler 5 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741924_1100{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 45 2016-10-20 14:34:32,806 DEBUG [RegionOpenAndInitThread-ns2:table2_restore-1] regionserver.HRegion(736): Instantiated ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:32,807 DEBUG [RegionOpenAndInitThread-ns2:table2_restore-1] regionserver.HRegion(1419): Closing ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8.: disabling compactions & flushes 2016-10-20 14:34:32,807 DEBUG [RegionOpenAndInitThread-ns2:table2_restore-1] regionserver.HRegion(1446): Updates disabled for region ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:32,807 INFO [RegionOpenAndInitThread-ns2:table2_restore-1] regionserver.HRegion(1552): Closed ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:32,962 DEBUG [ProcedureExecutor-20] hbase.MetaTableAccessor(1374): Put{"totalColumns":1,"row":"ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8.","families":{"info":[{"qualifier":"regioninfo","vlen":44,"tag":[],"timestamp":9223372036854775807}]}} 2016-10-20 14:34:32,966 INFO [ProcedureExecutor-20] hbase.MetaTableAccessor(1571): Added 1 2016-10-20 14:34:33,119 INFO [ProcedureExecutor-20] master.AssignmentManager(726): Assigning 1 region(s) to cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:33,120 ERROR [ProcedureExecutor-20] master.TableStateManager(134): Unable to get table ns2:table2_restore state org.apache.hadoop.hbase.TableNotFoundException: ns2:table2_restore at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1567) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1546) at org.apache.hadoop.hbase.util.ModifyRegionUtils.assignRegions(ModifyRegionUtils.java:254) at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.assignRegions(CreateTableProcedure.java:430) at org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure.executeFromState(TruncateTableProcedure.java:122) at org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure.executeFromState(TruncateTableProcedure.java:47) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:452) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1066) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:855) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:808) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:494) 2016-10-20 14:34:33,121 INFO [ProcedureExecutor-20] master.RegionStates(1106): Transition {81900e9262bb3c77f4b3fd7b6387aeb8 state=OFFLINE, ts=1476974073119, server=null} to {81900e9262bb3c77f4b3fd7b6387aeb8 state=PENDING_OPEN, ts=1476974073121, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:33,121 INFO [ProcedureExecutor-20] master.RegionStateStore(207): Updating hbase:meta row ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. with state=PENDING_OPEN, sn=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:33,124 INFO [PriorityRpcServer.handler=4,queue=0,port=37560] regionserver.RSRpcServices(1666): Open ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:33,135 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(6339): Opening region: {ENCODED => 81900e9262bb3c77f4b3fd7b6387aeb8, NAME => 'ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8.', STARTKEY => '', ENDKEY => ''} 2016-10-20 14:34:33,136 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table table2_restore 81900e9262bb3c77f4b3fd7b6387aeb8 2016-10-20 14:34:33,136 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(736): Instantiated ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:33,142 INFO [StoreOpener-81900e9262bb3c77f4b3fd7b6387aeb8-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:33,142 INFO [StoreOpener-81900e9262bb3c77f4b3fd7b6387aeb8-1] compactions.CompactionConfiguration(137): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, base window in milliseconds 21600000, windows per tier 4,incoming window min 6 2016-10-20 14:34:33,144 DEBUG [StoreOpener-81900e9262bb3c77f4b3fd7b6387aeb8-1] regionserver.HRegionFileSystem(202): No StoreFiles for: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/f 2016-10-20 14:34:33,146 DEBUG [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(3881): Found 0 recovered edits file(s) under hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8 2016-10-20 14:34:33,156 DEBUG [RS_OPEN_REGION-cn012:37560-2] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-10-20 14:34:33,156 INFO [RS_OPEN_REGION-cn012:37560-2] regionserver.HRegion(871): Onlined 81900e9262bb3c77f4b3fd7b6387aeb8; next sequenceid=2 2016-10-20 14:34:33,159 INFO [PostOpenDeployTasks:81900e9262bb3c77f4b3fd7b6387aeb8] regionserver.HRegionServer(1957): Post open deploy tasks for ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:33,160 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.AssignmentManager(2884): Got transition OPENED for {81900e9262bb3c77f4b3fd7b6387aeb8 state=PENDING_OPEN, ts=1476974073121, server=cn012.l42scl.hortonworks.com,37560,1476973974890} from cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:33,160 INFO [B.defaultRpcServer.handler=2,queue=0,port=41289] master.RegionStates(1106): Transition {81900e9262bb3c77f4b3fd7b6387aeb8 state=PENDING_OPEN, ts=1476974073121, server=cn012.l42scl.hortonworks.com,37560,1476973974890} to {81900e9262bb3c77f4b3fd7b6387aeb8 state=OPEN, ts=1476974073160, server=cn012.l42scl.hortonworks.com,37560,1476973974890} 2016-10-20 14:34:33,160 INFO [B.defaultRpcServer.handler=2,queue=0,port=41289] master.RegionStateStore(207): Updating hbase:meta row ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. with state=OPEN, openSeqNum=2, server=cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:33,162 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=41289] master.RegionStates(452): Onlined 81900e9262bb3c77f4b3fd7b6387aeb8 on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:33,163 DEBUG [ProcedureExecutor-20] master.AssignmentManager(897): Bulk assigning done for cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:33,163 DEBUG [ProcedureExecutor-20] hbase.MetaTableAccessor(1355): Put{"totalColumns":1,"row":"ns2:table2_restore","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1476974073163}]}} 2016-10-20 14:34:33,163 ERROR [B.defaultRpcServer.handler=2,queue=0,port=41289] master.TableStateManager(134): Unable to get table ns2:table2_restore state org.apache.hadoop.hbase.TableNotFoundException: ns2:table2_restore at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionOpen(AssignmentManager.java:2311) at org.apache.hadoop.hbase.master.AssignmentManager.onRegionTransition(AssignmentManager.java:2891) at org.apache.hadoop.hbase.master.MasterRpcServices.reportRegionStateTransition(MasterRpcServices.java:1350) at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8623) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2229) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:136) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:111) at java.lang.Thread.run(Thread.java:745) 2016-10-20 14:34:33,164 DEBUG [PostOpenDeployTasks:81900e9262bb3c77f4b3fd7b6387aeb8] regionserver.HRegionServer(1984): Finished post open deploy task for ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:33,169 DEBUG [RS_OPEN_REGION-cn012:37560-2] handler.OpenRegionHandler(126): Opened ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. on cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:33,172 INFO [ProcedureExecutor-20] hbase.MetaTableAccessor(1700): Updated table ns2:table2_restore state to ENABLED in META 2016-10-20 14:34:33,319 DEBUG [ProcedureExecutor-20] procedure.TruncateTableProcedure(129): truncate 'ns2:table2_restore' completed 2016-10-20 14:34:33,477 DEBUG [ProcedureExecutor-20] lock.ZKInterProcessLockBase(328): Released /1/table-lock/ns2:table2_restore/write-master:412890000000002 2016-10-20 14:34:33,478 DEBUG [ProcedureExecutor-20] procedure2.ProcedureExecutor(870): Procedure completed in 1.6090sec: TruncateTableProcedure (table=ns2:table2_restore preserveSplits=true) id=21 owner=hbase state=FINISHED 2016-10-20 14:34:33,614 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=41289] master.MasterRpcServices(974): Checking to see if procedure is done procId=21 2016-10-20 14:34:33,614 INFO [main] client.HBaseAdmin$TableFuture(3337): Operation: TRUNCATE, Table Name: ns2:table2_restore completed 2016-10-20 14:34:33,615 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@3dddefd8] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:38642 to delete [blk_1073741909_1085, blk_1073741910_1086] 2016-10-20 14:34:33,619 DEBUG [main] util.RestoreServerUtil(339): cluster hold the backup image: hdfs://localhost:37879; local cluster node: hdfs://localhost:37879 2016-10-20 14:34:33,619 DEBUG [main] util.RestoreServerUtil(345): File hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901/archive/data/ns2/test-14769740038901 on local cluster, back it up before restore 2016-10-20 14:34:33,622 INFO [IPC Server handler 0 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741915_1091 127.0.0.1:38642 2016-10-20 14:34:33,622 INFO [IPC Server handler 0 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741916_1092 127.0.0.1:38642 2016-10-20 14:34:33,649 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741925_1101{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 8292 2016-10-20 14:34:34,051 DEBUG [main] util.RestoreServerUtil(355): Copied to temporary path on local cluster: /user/hbase/hbase-staging/restore 2016-10-20 14:34:34,053 DEBUG [main] util.RestoreServerUtil(463): TableArchivePath for bulkload using tempPath: /user/hbase/hbase-staging/restore 2016-10-20 14:34:34,053 DEBUG [main] mapreduce.MapReduceRestoreTask(68): Restore full backup from directory hdfs://localhost:37879/user/hbase/hbase-staging/restore/20fb71680e16fdda348e8695b59a8391 from hbase tables ,ns2:test-14769740038901 to tables ,ns2:table2_restore 2016-10-20 14:34:34,053 INFO [main] mapreduce.MapReduceRestoreTask(76): Restore ns2:test-14769740038901 into ns2:table2_restore 2016-10-20 14:34:34,056 DEBUG [main] mapreduce.HFileSplitter(125): add incremental job :/user/hbase/hbase-staging/bulk_output-ns2-table2_restore-1476974074054 from hdfs://localhost:37879/user/hbase/hbase-staging/restore/20fb71680e16fdda348e8695b59a8391 2016-10-20 14:34:34,058 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x5ae15 connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:34:34,070 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x5ae150x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:34:34,073 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x5ae15-0x157e2823ec60019 connected 2016-10-20 14:34:34,073 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@577536e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:34:34,074 DEBUG [main] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:34:34,074 DEBUG [main] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:34:34,079 DEBUG [AsyncRpcChannel-pool2-t38] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service MasterService, sasl=false 2016-10-20 14:34:34,080 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:43048; # active connections: 8 2016-10-20 14:34:34,083 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:34,083 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 43048 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:34,086 INFO [main] mapreduce.HFileOutputFormat2(478): bulkload locality sensitive enabled 2016-10-20 14:34:34,087 INFO [main] mapreduce.HFileOutputFormat2(483): Looking up current regions for table ns2:test-14769740038901 2016-10-20 14:34:34,111 DEBUG [AsyncRpcChannel-pool2-t39] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:34,111 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:43050; # active connections: 9 2016-10-20 14:34:34,114 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:34,114 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 43050 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:34,121 INFO [main] mapreduce.HFileOutputFormat2(485): Configuring 1 reduce partitions to match current region count 2016-10-20 14:34:34,122 INFO [main] mapreduce.HFileOutputFormat2(378): Writing partition information to /user/hbase/hbase-staging/partitions_f84108d0-ee03-4078-aaee-b75595bf2f19 2016-10-20 14:34:34,132 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741926_1102{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:34:34,134 WARN [main] mapreduce.TableMapReduceUtil(786): The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it. 2016-10-20 14:34:34,305 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.HConstants, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-3810069950126074752.jar 2016-10-20 14:34:35,180 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-5978140196207042721.jar 2016-10-20 14:34:35,422 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.client.Put, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-5033773075231980237.jar 2016-10-20 14:34:35,436 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.CompatibilityFactory, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-657062112585243228.jar 2016-10-20 14:34:36,281 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-8738582648742289282.jar 2016-10-20 14:34:36,282 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.zookeeper.ZooKeeper, using jar /home/hbase/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar 2016-10-20 14:34:36,282 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class io.netty.channel.Channel, using jar /home/hbase/.m2/repository/io/netty/netty-all/4.0.30.Final/netty-all-4.0.30.Final.jar 2016-10-20 14:34:36,282 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.protobuf.Message, using jar /home/hbase/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar 2016-10-20 14:34:36,283 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.common.collect.Lists, using jar /home/hbase/.m2/repository/com/google/guava/guava/12.0.1/guava-12.0.1.jar 2016-10-20 14:34:36,283 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.htrace.Trace, using jar /home/hbase/.m2/repository/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar 2016-10-20 14:34:36,283 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.codahale.metrics.MetricRegistry, using jar /home/hbase/.m2/repository/io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.jar 2016-10-20 14:34:36,422 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-2949608787152221643.jar 2016-10-20 14:34:36,422 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.KeyValue, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-2949608787152221643.jar 2016-10-20 14:34:36,598 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-jobhistoryserver.properties,hadoop-metrics2.properties 2016-10-20 14:34:36,616 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@3dddefd8] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:38642 to delete [blk_1073741915_1091, blk_1073741916_1092] 2016-10-20 14:34:37,246 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.mapreduce.HFileInputFormat2, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-1362307258149324132.jar 2016-10-20 14:34:37,247 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-2949608787152221643.jar 2016-10-20 14:34:37,247 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.KeyValue, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-2949608787152221643.jar 2016-10-20 14:34:37,248 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-1362307258149324132.jar 2016-10-20 14:34:37,248 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.1/hadoop-mapreduce-client-core-2.7.1.jar 2016-10-20 14:34:37,249 INFO [main] mapreduce.HFileOutputFormat2(498): Incremental table ns2:test-14769740038901 output configured. 2016-10-20 14:34:37,249 INFO [main] client.ConnectionImplementation(1805): Closing master protocol: MasterService 2016-10-20 14:34:37,249 INFO [main] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec60019 2016-10-20 14:34:37,269 DEBUG [main] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:37,274 DEBUG [main] mapreduce.HFileSplitter(137): success configuring load incremental job 2016-10-20 14:34:37,274 DEBUG [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:43048 because read count=-1. Number of active connections: 9 2016-10-20 14:34:37,274 DEBUG [AsyncRpcChannel-pool2-t38] ipc.AsyncRpcChannel$8(566): IPC Client (745364024) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:37,274 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:43050 because read count=-1. Number of active connections: 9 2016-10-20 14:34:37,274 DEBUG [AsyncRpcChannel-pool2-t39] ipc.AsyncRpcChannel$8(566): IPC Client (1448740308) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:37,274 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.common.base.Preconditions, using jar /home/hbase/.m2/repository/com/google/guava/guava/12.0.1/guava-12.0.1.jar 2016-10-20 14:34:37,289 WARN [main] mapreduce.JobResourceUploader(64): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-10-20 14:34:37,308 WARN [main] mapreduce.JobResourceUploader(171): No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-10-20 14:34:38,687 INFO [LocalJobRunner Map Task Executor #0] mapreduce.HFileInputFormat2$HFileRecordReader(88): Initialize HFileRecordReader for hdfs://localhost:37879/user/hbase/hbase-staging/restore/20fb71680e16fdda348e8695b59a8391/f/a39b76fbbc004d008400803cfd71b6e0 2016-10-20 14:34:38,687 INFO [LocalJobRunner Map Task Executor #0] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:38,695 INFO [LocalJobRunner Map Task Executor #0] mapreduce.HFileInputFormat2$HFileRecordReader(103): Seeking to start 2016-10-20 14:34:38,741 INFO [pool-202-thread-1] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x401bceda connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:34:38,761 DEBUG [pool-202-thread-1-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x401bceda0x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:34:38,764 DEBUG [pool-202-thread-1-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x401bceda-0x157e2823ec6001a connected 2016-10-20 14:34:38,764 DEBUG [pool-202-thread-1] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@671d6fbc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:34:38,765 DEBUG [pool-202-thread-1] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:34:38,765 DEBUG [pool-202-thread-1] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:34:38,770 DEBUG [AsyncRpcChannel-pool2-t40] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:38,770 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:43070; # active connections: 8 2016-10-20 14:34:38,772 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:38,773 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 43070 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:38,776 INFO [pool-202-thread-1] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec6001a 2016-10-20 14:34:38,786 DEBUG [pool-202-thread-1] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:38,790 DEBUG [pool-202-thread-1] mapreduce.HFileOutputFormat2$1(232): first rowkey: [row0] 2016-10-20 14:34:38,790 DEBUG [AsyncRpcChannel-pool2-t40] ipc.AsyncRpcChannel$8(566): IPC Client (208835513) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:38,790 DEBUG [pool-202-thread-1] mapreduce.HFileOutputFormat2$1(244): use favored nodes writer: cn012.l42scl.hortonworks.com 2016-10-20 14:34:38,790 DEBUG [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:43070 because read count=-1. Number of active connections: 8 2016-10-20 14:34:38,791 INFO [pool-202-thread-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:38,810 WARN [Thread-2611] hdfs.DFSOutputStream$DataStreamer(1413): These favored nodes were specified but not chosen: [cn012.l42scl.hortonworks.com:37560] Specified favored nodes: [cn012.l42scl.hortonworks.com:37560] 2016-10-20 14:34:38,814 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741927_1103{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 8397 2016-10-20 14:34:39,694 DEBUG [main] mapreduce.MapReduceRestoreTask(93): Restoring HFiles from directory /user/hbase/hbase-staging/bulk_output-ns2-table2_restore-1476974074054 2016-10-20 14:34:39,695 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x2282400e connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:34:39,711 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x2282400e0x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:34:39,715 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x2282400e-0x157e2823ec6001b connected 2016-10-20 14:34:39,715 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@543d5863, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:34:39,715 DEBUG [main] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:34:39,715 DEBUG [main] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:34:39,719 DEBUG [AsyncRpcChannel-pool2-t41] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:39,719 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:43078; # active connections: 8 2016-10-20 14:34:39,723 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:39,723 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 43078 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:39,750 DEBUG [main] client.ConnectionImplementation(604): Table ns2:table2_restore should be available 2016-10-20 14:34:39,753 WARN [main] mapreduce.LoadIncrementalHFiles(202): Skipping non-directory hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns2-table2_restore-1476974074054/_SUCCESS 2016-10-20 14:34:39,763 DEBUG [AsyncRpcChannel-pool2-t42] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service MasterService, sasl=false 2016-10-20 14:34:39,764 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:43080; # active connections: 9 2016-10-20 14:34:39,767 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:39,767 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 43080 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:39,776 INFO [LoadIncrementalHFiles-0] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=6, currentSize=1090168, freeSize=1042872136, maxSize=1043962304, heapSize=1090168, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-10-20 14:34:39,782 INFO [LoadIncrementalHFiles-0] mapreduce.LoadIncrementalHFiles(820): Trying to load hfile=hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns2-table2_restore-1476974074054/f/0c553e50841846d5acf3e9a97992491d first=row0 last=row98 2016-10-20 14:34:39,790 DEBUG [LoadIncrementalHFiles-1] mapreduce.LoadIncrementalHFiles$4(913): Going to connect to server region=ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8., hostname=cn012.l42scl.hortonworks.com,37560,1476973974890, seqNum=2 for row with hfile group [{[B@5efeb117,hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns2-table2_restore-1476974074054/f/0c553e50841846d5acf3e9a97992491d}] 2016-10-20 14:34:39,795 DEBUG [AsyncRpcChannel-pool2-t43] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:39,795 DEBUG [RpcServer.listener,port=37560] ipc.RpcServer$Listener(880): RpcServer.listener,port=37560: connection from 172.18.128.12:42450; # active connections: 7 2016-10-20 14:34:39,798 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:39,798 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 42450 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:39,799 INFO [B.defaultRpcServer.handler=2,queue=0,port=37560] regionserver.HStore(670): Validating hfile at hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns2-table2_restore-1476974074054/f/0c553e50841846d5acf3e9a97992491d for inclusion in store f region ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:39,803 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=37560] regionserver.HStore(682): HFile bounds: first=row0 last=row98 2016-10-20 14:34:39,803 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=37560] regionserver.HStore(684): Region bounds: first= last= 2016-10-20 14:34:39,808 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=37560] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns2-table2_restore-1476974074054/f/0c553e50841846d5acf3e9a97992491d as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/f/9243c3b8bf8d44539aceb1b816a30f2a_SeqId_4_ 2016-10-20 14:34:39,809 INFO [B.defaultRpcServer.handler=2,queue=0,port=37560] regionserver.HStore(742): Loaded HFile hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns2-table2_restore-1476974074054/f/0c553e50841846d5acf3e9a97992491d into store 'f' as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/f/9243c3b8bf8d44539aceb1b816a30f2a_SeqId_4_ - updating store file list. 2016-10-20 14:34:39,816 INFO [B.defaultRpcServer.handler=2,queue=0,port=37560] regionserver.HStore(777): Loaded HFile hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/f/9243c3b8bf8d44539aceb1b816a30f2a_SeqId_4_ into store 'f 2016-10-20 14:34:39,817 INFO [B.defaultRpcServer.handler=2,queue=0,port=37560] regionserver.HStore(748): Successfully loaded store file hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns2-table2_restore-1476974074054/f/0c553e50841846d5acf3e9a97992491d into store f (new location: hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/f/9243c3b8bf8d44539aceb1b816a30f2a_SeqId_4_) 2016-10-20 14:34:39,819 INFO [main] client.ConnectionImplementation(1805): Closing master protocol: MasterService 2016-10-20 14:34:39,820 INFO [main] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec6001b 2016-10-20 14:34:39,827 DEBUG [main] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:39,830 DEBUG [main] mapreduce.MapReduceRestoreTask(106): Restore Job finished:0 2016-10-20 14:34:39,830 DEBUG [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Listener(912): RpcServer.listener,port=37560: DISCONNECTING client 172.18.128.12:42450 because read count=-1. Number of active connections: 7 2016-10-20 14:34:39,830 DEBUG [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:43078 because read count=-1. Number of active connections: 9 2016-10-20 14:34:39,830 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:43080 because read count=-1. Number of active connections: 9 2016-10-20 14:34:39,830 INFO [main] impl.RestoreTablesClient(168): Restoring 'ns2:test-14769740038901' to 'ns2:table2_restore' from log dirs: hdfs://localhost:37879/backupUT/backup_1476974042777/WALs 2016-10-20 14:34:39,830 DEBUG [AsyncRpcChannel-pool2-t43] ipc.AsyncRpcChannel$8(566): IPC Client (1410166625) to cn012.l42scl.hortonworks.com/172.18.128.12:37560 from hbase: closed 2016-10-20 14:34:39,830 DEBUG [AsyncRpcChannel-pool2-t42] ipc.AsyncRpcChannel$8(566): IPC Client (-1547483546) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:39,830 DEBUG [AsyncRpcChannel-pool2-t41] ipc.AsyncRpcChannel$8(566): IPC Client (-2086032703) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:39,837 DEBUG [main] util.RestoreServerUtil(219): Found descriptor 'ns2:test-14769740038901', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} through backup_1476974042777 2016-10-20 14:34:39,840 DEBUG [main] mapreduce.MapReduceRestoreTask(68): Restore incremental backup from directory hdfs://localhost:37879/backupUT/backup_1476974042777/WALs from hbase tables ,ns2:test-14769740038901 to tables ,ns2:table2_restore 2016-10-20 14:34:39,840 INFO [main] mapreduce.MapReduceRestoreTask(76): Restore ns2:test-14769740038901 into ns2:table2_restore 2016-10-20 14:34:39,842 DEBUG [main] mapreduce.WALPlayer(299): add incremental job :/user/hbase/hbase-staging/bulk_output-ns2-table2_restore-1476974079840 from hdfs://localhost:37879/backupUT/backup_1476974042777/WALs 2016-10-20 14:34:39,843 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x4f281aaf connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:34:39,852 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x4f281aaf0x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:34:39,853 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d508d22, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:34:39,854 DEBUG [main] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:34:39,856 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x4f281aaf-0x157e2823ec6001c connected 2016-10-20 14:34:39,856 DEBUG [main] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:34:39,858 DEBUG [AsyncRpcChannel-pool2-t44] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service MasterService, sasl=false 2016-10-20 14:34:39,858 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:43088; # active connections: 8 2016-10-20 14:34:39,861 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:39,861 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 43088 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:39,864 INFO [main] mapreduce.HFileOutputFormat2(478): bulkload locality sensitive enabled 2016-10-20 14:34:39,864 INFO [main] mapreduce.HFileOutputFormat2(483): Looking up current regions for table ns2:test-14769740038901 2016-10-20 14:34:39,873 DEBUG [AsyncRpcChannel-pool2-t45] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:39,873 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:43090; # active connections: 9 2016-10-20 14:34:39,875 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:39,876 INFO [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 43090 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:39,881 INFO [main] mapreduce.HFileOutputFormat2(485): Configuring 1 reduce partitions to match current region count 2016-10-20 14:34:39,882 INFO [main] mapreduce.HFileOutputFormat2(378): Writing partition information to /user/hbase/hbase-staging/partitions_38dda431-3db1-490d-953d-1c2519b9e2e2 2016-10-20 14:34:39,891 INFO [IPC Server handler 4 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741928_1104{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 153 2016-10-20 14:34:40,295 WARN [main] mapreduce.TableMapReduceUtil(786): The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it. 2016-10-20 14:34:40,489 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.HConstants, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-8361213880808926341.jar 2016-10-20 14:34:41,470 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-8303354739380982454.jar 2016-10-20 14:34:41,843 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.client.Put, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-5702470428881046087.jar 2016-10-20 14:34:41,860 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.CompatibilityFactory, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-5482223174396804425.jar 2016-10-20 14:34:43,042 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-5613228494153924827.jar 2016-10-20 14:34:43,042 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.zookeeper.ZooKeeper, using jar /home/hbase/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar 2016-10-20 14:34:43,043 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class io.netty.channel.Channel, using jar /home/hbase/.m2/repository/io/netty/netty-all/4.0.30.Final/netty-all-4.0.30.Final.jar 2016-10-20 14:34:43,043 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.protobuf.Message, using jar /home/hbase/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar 2016-10-20 14:34:43,043 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.common.collect.Lists, using jar /home/hbase/.m2/repository/com/google/guava/guava/12.0.1/guava-12.0.1.jar 2016-10-20 14:34:43,044 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.htrace.Trace, using jar /home/hbase/.m2/repository/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar 2016-10-20 14:34:43,044 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.codahale.metrics.MetricRegistry, using jar /home/hbase/.m2/repository/io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.jar 2016-10-20 14:34:43,209 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-7050814894701059432.jar 2016-10-20 14:34:43,209 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.KeyValue, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-7050814894701059432.jar 2016-10-20 14:34:44,181 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.mapreduce.WALInputFormat, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-1441761547527093357.jar 2016-10-20 14:34:44,182 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-7050814894701059432.jar 2016-10-20 14:34:44,182 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.KeyValue, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-7050814894701059432.jar 2016-10-20 14:34:44,182 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2, using jar /mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/hadoop-1441761547527093357.jar 2016-10-20 14:34:44,183 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner, using jar /home/hbase/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.1/hadoop-mapreduce-client-core-2.7.1.jar 2016-10-20 14:34:44,183 INFO [main] mapreduce.HFileOutputFormat2(498): Incremental table ns2:test-14769740038901 output configured. 2016-10-20 14:34:44,183 INFO [main] client.ConnectionImplementation(1805): Closing master protocol: MasterService 2016-10-20 14:34:44,183 INFO [main] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec6001c 2016-10-20 14:34:44,194 DEBUG [main] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:44,199 DEBUG [main] mapreduce.WALPlayer(316): success configuring load incremental job 2016-10-20 14:34:44,199 DEBUG [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:43090 because read count=-1. Number of active connections: 9 2016-10-20 14:34:44,199 DEBUG [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:43088 because read count=-1. Number of active connections: 9 2016-10-20 14:34:44,199 DEBUG [AsyncRpcChannel-pool2-t45] ipc.AsyncRpcChannel$8(566): IPC Client (1994526019) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:44,199 DEBUG [AsyncRpcChannel-pool2-t44] ipc.AsyncRpcChannel$8(566): IPC Client (-632606392) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:44,200 DEBUG [main] mapreduce.TableMapReduceUtil(920): For class com.google.common.base.Preconditions, using jar /home/hbase/.m2/repository/com/google/guava/guava/12.0.1/guava-12.0.1.jar 2016-10-20 14:34:44,216 WARN [main] mapreduce.JobResourceUploader(64): Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2016-10-20 14:34:44,235 WARN [main] mapreduce.JobResourceUploader(171): No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2016-10-20 14:34:44,650 DEBUG [main] mapreduce.WALInputFormat(265): Scanning hdfs://localhost:37879/backupUT/backup_1476974042777/WALs for WAL files 2016-10-20 14:34:44,653 WARN [main] mapreduce.WALInputFormat(289): File hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/.backup.manifest does not appear to be an WAL file. Skipping... 2016-10-20 14:34:44,654 INFO [main] mapreduce.WALInputFormat(281): Found: LocatedFileStatus{path=hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974036606; isDirectory=false; length=1734; replication=1; blocksize=134217728; modification_time=1476974045719; access_time=1476974045304; owner=hbase; group=supergroup; permission=rw-r--r--; isSymlink=false} 2016-10-20 14:34:44,654 INFO [main] mapreduce.WALInputFormat(281): Found: LocatedFileStatus{path=hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974037083; isDirectory=false; length=5239; replication=1; blocksize=134217728; modification_time=1476974045753; access_time=1476974045741; owner=hbase; group=supergroup; permission=rw-r--r--; isSymlink=false} 2016-10-20 14:34:44,654 INFO [main] mapreduce.WALInputFormat(281): Found: LocatedFileStatus{path=hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974036606; isDirectory=false; length=91; replication=1; blocksize=134217728; modification_time=1476974045782; access_time=1476974045771; owner=hbase; group=supergroup; permission=rw-r--r--; isSymlink=false} 2016-10-20 14:34:44,654 INFO [main] mapreduce.WALInputFormat(281): Found: LocatedFileStatus{path=hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974036670; isDirectory=false; length=91; replication=1; blocksize=134217728; modification_time=1476974045812; access_time=1476974045801; owner=hbase; group=supergroup; permission=rw-r--r--; isSymlink=false} 2016-10-20 14:34:45,763 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(152): Opening reader for hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974037083 (-9223372036854775808:9223372036854775807) length:5239 2016-10-20 14:34:45,774 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(185): Reached end of file. 2016-10-20 14:34:45,775 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(210): Closing reader 2016-10-20 14:34:45,813 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(152): Opening reader for hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974036606 (-9223372036854775808:9223372036854775807) length:1734 2016-10-20 14:34:45,819 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(185): Reached end of file. 2016-10-20 14:34:45,819 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(210): Closing reader 2016-10-20 14:34:45,841 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(152): Opening reader for hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974036606 (-9223372036854775808:9223372036854775807) length:91 2016-10-20 14:34:45,846 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(185): Reached end of file. 2016-10-20 14:34:45,846 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(210): Closing reader 2016-10-20 14:34:45,865 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(152): Opening reader for hdfs://localhost:37879/backupUT/backup_1476974042777/WALs/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974036670 (-9223372036854775808:9223372036854775807) length:91 2016-10-20 14:34:45,869 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(185): Reached end of file. 2016-10-20 14:34:45,869 INFO [LocalJobRunner Map Task Executor #0] mapreduce.WALInputFormat$WALRecordReader(210): Closing reader 2016-10-20 14:34:46,765 DEBUG [main] mapreduce.MapReduceRestoreTask(93): Restoring HFiles from directory /user/hbase/hbase-staging/bulk_output-ns2-table2_restore-1476974079840 2016-10-20 14:34:46,766 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x5bde57ab connecting to ZooKeeper ensemble=localhost:56750 2016-10-20 14:34:46,795 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x5bde57ab0x0, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-10-20 14:34:46,798 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(674): hconnection-0x5bde57ab-0x157e2823ec6001d connected 2016-10-20 14:34:46,798 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ea00de, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-10-20 14:34:46,799 DEBUG [main] ipc.AsyncRpcClient(160): Starting async Hbase RPC client 2016-10-20 14:34:46,799 DEBUG [main] ipc.AsyncRpcClient(171): Use global event loop group NioEventLoopGroup 2016-10-20 14:34:46,803 DEBUG [AsyncRpcChannel-pool2-t46] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service ClientService, sasl=false 2016-10-20 14:34:46,803 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:43120; # active connections: 8 2016-10-20 14:34:46,807 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:46,807 INFO [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 43120 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:46,835 DEBUG [main] client.ConnectionImplementation(604): Table ns2:table2_restore should be available 2016-10-20 14:34:46,839 WARN [main] mapreduce.LoadIncrementalHFiles(202): Skipping non-directory hdfs://localhost:37879/user/hbase/hbase-staging/bulk_output-ns2-table2_restore-1476974079840/_SUCCESS 2016-10-20 14:34:46,842 DEBUG [AsyncRpcChannel-pool2-t47] ipc.AsyncRpcChannel(479): Use SIMPLE authentication for service MasterService, sasl=false 2016-10-20 14:34:46,842 DEBUG [RpcServer.listener,port=41289] ipc.RpcServer$Listener(880): RpcServer.listener,port=41289: connection from 172.18.128.12:43122; # active connections: 9 2016-10-20 14:34:46,847 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1710): Auth successful for hbase (auth:SIMPLE) 2016-10-20 14:34:46,847 INFO [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Connection(1740): Connection from 172.18.128.12 port: 43122 with version info: version: "2.0.0-SNAPSHOT" url: "git://cn012.l42scl.hortonworks.com/mnt/disk2/a/7912" revision: "9dd28f2a944b15932c61728ccba2458687d6439e" user: "hbase" date: "Thu Oct 20 14:16:08 UTC 2016" src_checksum: "7c2a651fa8a7403e988d0a940fa8c91f" version_major: 2 version_minor: 0 2016-10-20 14:34:46,850 WARN [main] mapreduce.LoadIncrementalHFiles(445): /user/hbase/hbase-staging/bulk_output-ns2-table2_restore-1476974079840 2016-10-20 14:34:46,851 INFO [main] client.ConnectionImplementation(1805): Closing master protocol: MasterService 2016-10-20 14:34:46,851 INFO [main] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec6001d 2016-10-20 14:34:46,861 DEBUG [main] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:46,864 DEBUG [main] mapreduce.MapReduceRestoreTask(106): Restore Job finished:0 2016-10-20 14:34:46,864 INFO [main] impl.RestoreTablesClient(173): ns2:test-14769740038901 has been successfully restored to ns2:table2_restore 2016-10-20 14:34:46,864 DEBUG [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:43122 because read count=-1. Number of active connections: 9 2016-10-20 14:34:46,864 INFO [main] impl.RestoreTablesClient(205): Restore includes the following image(s): 2016-10-20 14:34:46,865 INFO [main] impl.RestoreTablesClient(207): Backup: backup_1476974021410 hdfs://localhost:37879/backupUT/backup_1476974021410/ns2/test-14769740038901/ 2016-10-20 14:34:46,865 DEBUG [main] impl.RestoreTablesClient(219): restoreStage finished 2016-10-20 14:34:46,864 DEBUG [AsyncRpcChannel-pool2-t46] ipc.AsyncRpcChannel$8(566): IPC Client (800029933) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:46,864 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:43120 because read count=-1. Number of active connections: 9 2016-10-20 14:34:46,864 DEBUG [AsyncRpcChannel-pool2-t47] ipc.AsyncRpcChannel$8(566): IPC Client (1938970379) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:46,868 DEBUG [main] backup.TestIncrementalBackup(194): After incremental restore: 'ns1:table1_restore', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2016-10-20 14:34:46,875 DEBUG [main] client.ClientScanner(279): Advancing internal scanner to startKey at 'row0' 2016-10-20 14:34:46,892 DEBUG [main] backup.TestIncrementalBackup(195): f1 has 99 rows 2016-10-20 14:34:46,902 DEBUG [main] client.ClientScanner(279): Advancing internal scanner to startKey at 'row0' 2016-10-20 14:34:46,992 INFO [main] hbase.ResourceChecker(172): after: backup.TestIncrementalBackup#TestIncBackupRestore[0] Thread=947 (was 871) Potentially hanging thread: ResponseProcessor for block BP-1003445509-172.18.128.12-1476973970079:blk_1073741892_1068 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) java.io.FilterInputStream.read(FilterInputStream.java:83) java.io.FilterInputStream.read(FilterInputStream.java:83) org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2278) org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:734) Potentially hanging thread: AM.-pool3-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t42 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x2dfb885e-shared-pool31-t213 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataStreamer for file /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1.1476974043279 block BP-1003445509-172.18.128.12-1476973970079:blk_1073741891_1067 java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:418) Potentially hanging thread: PacketResponder: BP-1003445509-172.18.128.12-1476973970079:blk_1073741890_1066, type=LAST_IN_PIPELINE, downstreams=0:[] java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1184) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1255) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x2dfb885e-shared-pool31-t211 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PacketResponder: BP-1003445509-172.18.128.12-1476973970079:blk_1073741889_1065, type=LAST_IN_PIPELINE, downstreams=0:[] java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1184) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1255) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS_CLOSE_REGION-cn012:37560-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x2dfb885e-shared-pool31-t212 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t40 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x2dfb885e-shared-pool31-t215 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x2dfb885e-shared-pool31-t209 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1897337416_1 at /127.0.0.1:49164 [Receiving block BP-1003445509-172.18.128.12-1476973970079:blk_1073741890_1066] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:849) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:804) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool19-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool28-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ResponseProcessor for block BP-1003445509-172.18.128.12-1476973970079:blk_1073741889_1065 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) java.io.FilterInputStream.read(FilterInputStream.java:83) java.io.FilterInputStream.read(FilterInputStream.java:83) org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2278) org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:734) Potentially hanging thread: Async disk worker #0 for volume /mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/dfscluster_6b2a65f6-e6e0-418a-b6b5-3f3b53b4377b/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataStreamer for file /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0.1476974042856 block BP-1003445509-172.18.128.12-1476973970079:blk_1073741889_1065 java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:418) Potentially hanging thread: AsyncRpcChannel-pool2-t11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t33 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t31 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t28 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x2dfb885e-shared-pool31-t210 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t20 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: MASTER_TABLE_OPERATIONS-cn012:41289-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: rs(cn012.l42scl.hortonworks.com,37560,1476973974890)-backup-pool25-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AM.-pool3-t4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t17 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_184315200_1 at /127.0.0.1:49162 [Receiving block BP-1003445509-172.18.128.12-1476973970079:blk_1073741889_1065] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:849) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:804) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t36 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_184315200_1 at /127.0.0.1:49168 [Receiving block BP-1003445509-172.18.128.12-1476973970079:blk_1073741892_1068] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:849) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:804) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t24 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t47 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t39 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool9-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t22 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t45 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool5-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1443929603_1 at /127.0.0.1:49494 [Waiting for operation #8] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t30 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS_CLOSE_REGION-cn012:37560-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t41 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t27 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t29 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t35 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: IPC Parameter Sending Thread #2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t26 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x2dfb885e-shared-pool31-t208 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: member: 'cn012.l42scl.hortonworks.com,41289,1476973974467' subprocedure-pool8-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: hconnection-0x2dfb885e-shared-pool31-t214 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: member: 'cn012.l42scl.hortonworks.com,37560,1476973974890' subprocedure-pool6-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t46 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t25 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t18 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ResponseProcessor for block BP-1003445509-172.18.128.12-1476973970079:blk_1073741890_1066 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) java.io.FilterInputStream.read(FilterInputStream.java:83) java.io.FilterInputStream.read(FilterInputStream.java:83) org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2278) org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:734) Potentially hanging thread: AsyncRpcChannel-pool2-t13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t34 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: (cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool7-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t32 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool27-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t37 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AM.-pool3-t2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool26-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PacketResponder: BP-1003445509-172.18.128.12-1476973970079:blk_1073741892_1068, type=LAST_IN_PIPELINE, downstreams=0:[] java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1184) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1255) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataStreamer for file /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467/cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1.1476974043284 block BP-1003445509-172.18.128.12-1476973970079:blk_1073741892_1068 java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:418) Potentially hanging thread: AsyncRpcChannel-pool2-t38 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1897337416_1 at /127.0.0.1:49166 [Receiving block BP-1003445509-172.18.128.12-1476973970079:blk_1073741891_1067] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:849) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:804) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: RS_CLOSE_REGION-cn012:37560-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: (cn012.l42scl.hortonworks.com,41289,1476973974467)-proc-coordinator-pool4-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t21 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t19 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: DataStreamer for file /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890/cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0.1476974042857 block BP-1003445509-172.18.128.12-1476973970079:blk_1073741890_1066 java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:418) Potentially hanging thread: rs(cn012.l42scl.hortonworks.com,41289,1476973974467)-backup-pool20-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t44 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: AsyncRpcChannel-pool2-t43 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Async disk worker #0 for volume /mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/dfscluster_6b2a65f6-e6e0-418a-b6b5-3f3b53b4377b/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: ResponseProcessor for block BP-1003445509-172.18.128.12-1476973970079:blk_1073741891_1067 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) java.io.FilterInputStream.read(FilterInputStream.java:83) java.io.FilterInputStream.read(FilterInputStream.java:83) org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2278) org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:734) Potentially hanging thread: AsyncRpcChannel-pool2-t23 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: PacketResponder: BP-1003445509-172.18.128.12-1476973970079:blk_1073741891_1067, type=LAST_IN_PIPELINE, downstreams=0:[] java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1184) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1255) java.lang.Thread.run(Thread.java:745) - Thread LEAK? -, OpenFileDescriptor=1820 (was 1695) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=32768 (was 32768), SystemLoadAverage=122 (was 76) - SystemLoadAverage LEAK? -, ProcessCount=337 (was 337), AvailableMemoryMB=85659 (was 86165) 2016-10-20 14:34:46,993 WARN [main] hbase.ResourceChecker(135): Thread=947 is superior to 500 2016-10-20 14:34:46,993 WARN [main] hbase.ResourceChecker(135): OpenFileDescriptor=1820 is superior to 1024 2016-10-20 14:34:47,016 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741907_1083 127.0.0.1:38642 2016-10-20 14:34:47,017 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741908_1084 127.0.0.1:38642 2016-10-20 14:34:47,017 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741919_1095 127.0.0.1:38642 2016-10-20 14:34:47,017 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741856_1032 127.0.0.1:38642 2016-10-20 14:34:47,018 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(1106): BLOCK* addToInvalidates: blk_1073741911_1087 127.0.0.1:38642 2016-10-20 14:34:47,018 INFO [main] hbase.HBaseTestingUtility(1143): Shutting down minicluster 2016-10-20 14:34:47,019 INFO [main] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec6000d 2016-10-20 14:34:47,027 DEBUG [main] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:47,028 DEBUG [main] util.JVMClusterUtil(241): Shutting down HBase Cluster 2016-10-20 14:34:47,028 DEBUG [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=34094] ipc.RpcServer$Listener(912): RpcServer.listener,port=34094: DISCONNECTING client 172.18.128.12:33114 because read count=-1. Number of active connections: 2 2016-10-20 14:34:47,028 DEBUG [AsyncRpcChannel-pool2-t5] ipc.AsyncRpcChannel$8(566): IPC Client (-860166644) to cn012.l42scl.hortonworks.com/172.18.128.12:34094 from hbase: closed 2016-10-20 14:34:47,028 DEBUG [main] coprocessor.CoprocessorHost(271): Stop coprocessor org.apache.hadoop.hbase.backup.master.BackupController 2016-10-20 14:34:47,031 INFO [main] regionserver.HRegionServer(1923): STOPPED: Cluster shutdown requested 2016-10-20 14:34:47,032 INFO [M:0;cn012:34094] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2016-10-20 14:34:47,032 INFO [SplitLogWorker-cn012:34094] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2016-10-20 14:34:47,032 INFO [SplitLogWorker-cn012:34094] regionserver.SplitLogWorker(155): SplitLogWorker cn012.l42scl.hortonworks.com,34094,1476973981831 exiting 2016-10-20 14:34:47,033 INFO [M:0;cn012:34094] regionserver.HeapMemoryManager(202): Stoping HeapMemoryTuner chore. 2016-10-20 14:34:47,033 INFO [M:0;cn012:34094] procedure2.ProcedureExecutor(532): Stopping the procedure executor 2016-10-20 14:34:47,036 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2016-10-20 14:34:47,036 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2016-10-20 14:34:47,036 INFO [M:0;cn012:34094] wal.WALProcedureStore(232): Stopping the WAL Procedure Store 2016-10-20 14:34:47,036 INFO [main] regionserver.HRegionServer(1923): STOPPED: Shutdown requested 2016-10-20 14:34:47,037 INFO [RS:0;cn012:34206] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2016-10-20 14:34:47,037 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/running 2016-10-20 14:34:47,037 INFO [RS:0;cn012:34206] regionserver.HeapMemoryManager(202): Stoping HeapMemoryTuner chore. 2016-10-20 14:34:47,037 INFO [SplitLogWorker-cn012:34206] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2016-10-20 14:34:47,037 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:34206-0x157e2823ec60008, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/running 2016-10-20 14:34:47,037 INFO [SplitLogWorker-cn012:34206] regionserver.SplitLogWorker(155): SplitLogWorker cn012.l42scl.hortonworks.com,34206,1476973982007 exiting 2016-10-20 14:34:47,037 INFO [RS:0;cn012:34206] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2016-10-20 14:34:47,037 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2016-10-20 14:34:47,041 INFO [RS:0;cn012:34206] regionserver.LogRollRegionServerProcedureManager(96): Stopping RegionServerBackupManager gracefully. 2016-10-20 14:34:47,040 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2016-10-20 14:34:47,041 INFO [RS:0;cn012:34206] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2016-10-20 14:34:47,038 DEBUG [main-EventThread] zookeeper.ZKUtil(367): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on znode that does not yet exist, /2/running 2016-10-20 14:34:47,038 DEBUG [main-EventThread] zookeeper.ZKUtil(367): regionserver:34206-0x157e2823ec60008, quorum=localhost:56750, baseZNode=/2 Set watcher on znode that does not yet exist, /2/running 2016-10-20 14:34:47,043 INFO [RS:0;cn012:34206] regionserver.HRegionServer(1063): stopping server cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:34:47,043 DEBUG [RS_CLOSE_REGION-cn012:34206-0] handler.CloseRegionHandler(90): Processing close of hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5. 2016-10-20 14:34:47,043 DEBUG [RS:0;cn012:34206] zookeeper.MetaTableLocator(612): Stopping MetaTableLocator 2016-10-20 14:34:47,045 DEBUG [RS_CLOSE_REGION-cn012:34206-0] regionserver.HRegion(1419): Closing hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5.: disabling compactions & flushes 2016-10-20 14:34:47,045 INFO [RS:0;cn012:34206] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec6000a 2016-10-20 14:34:47,046 DEBUG [RS_CLOSE_REGION-cn012:34206-0] regionserver.HRegion(1446): Updates disabled for region hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5. 2016-10-20 14:34:47,046 INFO [StoreCloserThread-hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5.-1] regionserver.HStore(839): Closed meta 2016-10-20 14:34:47,046 INFO [StoreCloserThread-hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5.-1] regionserver.HStore(839): Closed session 2016-10-20 14:34:47,052 DEBUG [RS:0;cn012:34206] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:47,055 INFO [RS:0;cn012:34206] regionserver.HRegionServer(1292): Waiting on 1 regions to close 2016-10-20 14:34:47,055 DEBUG [RS:0;cn012:34206] regionserver.HRegionServer(1296): {4a6cefc1b47013b7aa8a0dd5a0a290e5=hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5.} 2016-10-20 14:34:47,063 INFO [IPC Server handler 4 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a82af92a-d58e-4076-affe-976f93711253:NORMAL:127.0.0.1:43453|RBW]]} size 469 2016-10-20 14:34:47,066 INFO [M:0;cn012:34094] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2016-10-20 14:34:47,067 INFO [M:0;cn012:34094] regionserver.LogRollRegionServerProcedureManager(96): Stopping RegionServerBackupManager gracefully. 2016-10-20 14:34:47,067 INFO [M:0;cn012:34094] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2016-10-20 14:34:47,067 INFO [M:0;cn012:34094] regionserver.HRegionServer(1063): stopping server cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:34:47,067 DEBUG [M:0;cn012:34094] zookeeper.MetaTableLocator(612): Stopping MetaTableLocator 2016-10-20 14:34:47,067 DEBUG [RS_CLOSE_REGION-cn012:34094-0] handler.CloseRegionHandler(90): Processing close of hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f. 2016-10-20 14:34:47,067 INFO [M:0;cn012:34094] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec60009 2016-10-20 14:34:47,069 DEBUG [RS_CLOSE_REGION-cn012:34094-0] regionserver.HRegion(1419): Closing hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f.: disabling compactions & flushes 2016-10-20 14:34:47,069 DEBUG [RS_CLOSE_REGION-cn012:34094-0] regionserver.HRegion(1446): Updates disabled for region hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f. 2016-10-20 14:34:47,069 INFO [RS_CLOSE_REGION-cn012:34094-0] regionserver.HRegion(2345): Flushing 1/1 column families, memstore=344 B 2016-10-20 14:34:47,070 DEBUG [RS_CLOSE_REGION-cn012:34206-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/backup/4a6cefc1b47013b7aa8a0dd5a0a290e5/recovered.edits/4.seqid to file, newSeqId=4, maxSeqId=2 2016-10-20 14:34:47,076 INFO [RS_CLOSE_REGION-cn012:34206-0] regionserver.HRegion(1552): Closed hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5. 2016-10-20 14:34:47,078 DEBUG [RS_CLOSE_REGION-cn012:34206-0] handler.CloseRegionHandler(122): Closed hbase:backup,,1476973987272.4a6cefc1b47013b7aa8a0dd5a0a290e5. 2016-10-20 14:34:47,078 DEBUG [M:0;cn012:34094] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:47,079 INFO [M:0;cn012:34094] regionserver.CompactSplitThread(403): Waiting for Split Thread to finish... 2016-10-20 14:34:47,079 INFO [M:0;cn012:34094] regionserver.CompactSplitThread(403): Waiting for Merge Thread to finish... 2016-10-20 14:34:47,079 INFO [M:0;cn012:34094] regionserver.CompactSplitThread(403): Waiting for Large Compaction Thread to finish... 2016-10-20 14:34:47,079 INFO [M:0;cn012:34094] regionserver.CompactSplitThread(403): Waiting for Small Compaction Thread to finish... 2016-10-20 14:34:47,079 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=34206] ipc.RpcServer$Listener(912): RpcServer.listener,port=34206: DISCONNECTING client 172.18.128.12:49270 because read count=-1. Number of active connections: 1 2016-10-20 14:34:47,079 DEBUG [AsyncRpcChannel-pool2-t6] ipc.AsyncRpcChannel$8(566): IPC Client (-407707540) to cn012.l42scl.hortonworks.com/172.18.128.12:34206 from hbase: closed 2016-10-20 14:34:47,081 INFO [M:0;cn012:34094] regionserver.HRegionServer(1292): Waiting on 2 regions to close 2016-10-20 14:34:47,081 DEBUG [M:0;cn012:34094] regionserver.HRegionServer(1296): {37fa3cc8df36677363de1aacf6de184f=hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f., 1588230740=hbase:meta,,1.1588230740} 2016-10-20 14:34:47,081 DEBUG [RS_CLOSE_META-cn012:34094-0] handler.CloseRegionHandler(90): Processing close of hbase:meta,,1.1588230740 2016-10-20 14:34:47,083 DEBUG [RS_CLOSE_META-cn012:34094-0] regionserver.HRegion(1419): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2016-10-20 14:34:47,083 DEBUG [RS_CLOSE_META-cn012:34094-0] regionserver.HRegion(1446): Updates disabled for region hbase:meta,,1.1588230740 2016-10-20 14:34:47,083 INFO [RS_CLOSE_META-cn012:34094-0] regionserver.HRegion(2345): Flushing 2/2 column families, memstore=4.08 KB 2016-10-20 14:34:47,092 INFO [IPC Server handler 3 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741839_1015{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-651c82c6-88ce-4f3b-bae6-e3460c66df10:NORMAL:127.0.0.1:43453|RBW]]} size 4912 2016-10-20 14:34:47,094 INFO [IPC Server handler 1 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741840_1016{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a82af92a-d58e-4076-affe-976f93711253:NORMAL:127.0.0.1:43453|RBW]]} size 6418 2016-10-20 14:34:47,255 INFO [RS:0;cn012:34206] regionserver.HRegionServer(1091): stopping server cn012.l42scl.hortonworks.com,34206,1476973982007; all regions closed. 2016-10-20 14:34:47,256 DEBUG [RS:0;cn012:34206] wal.FSHLog(1085): Closing WAL writer in /user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:34:47,262 INFO [IPC Server handler 2 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-651c82c6-88ce-4f3b-bae6-e3460c66df10:NORMAL:127.0.0.1:43453|RBW]]} size 83 2016-10-20 14:34:47,268 DEBUG [RS:0;cn012:34206] wal.FSHLog(1043): Moved 1 WAL file(s) to /user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/oldWALs 2016-10-20 14:34:47,269 INFO [RS:0;cn012:34206] wal.FSHLog(1046): Closed WAL: FSHLog cn012.l42scl.hortonworks.com%2C34206%2C1476973982007.regiongroup-0:(num 1476973985825) 2016-10-20 14:34:47,269 DEBUG [RS:0;cn012:34206] wal.FSHLog(1085): Closing WAL writer in /user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:34:47,275 INFO [IPC Server handler 8 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741838_1014{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a82af92a-d58e-4076-affe-976f93711253:NORMAL:127.0.0.1:43453|RBW]]} size 83 2016-10-20 14:34:47,279 DEBUG [RS:0;cn012:34206] wal.FSHLog(1043): Moved 1 WAL file(s) to /user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/oldWALs 2016-10-20 14:34:47,280 INFO [RS:0;cn012:34206] wal.FSHLog(1046): Closed WAL: FSHLog cn012.l42scl.hortonworks.com%2C34206%2C1476973982007.regiongroup-1:(num 1476973988041) 2016-10-20 14:34:47,280 DEBUG [RS:0;cn012:34206] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:47,280 INFO [RS:0;cn012:34206] regionserver.Leases(146): RS:0;cn012:34206 closing leases 2016-10-20 14:34:47,280 INFO [RS:0;cn012:34206] regionserver.Leases(149): RS:0;cn012:34206 closed leases 2016-10-20 14:34:47,280 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=34094] ipc.RpcServer$Listener(912): RpcServer.listener,port=34094: DISCONNECTING client 172.18.128.12:41575 because read count=-1. Number of active connections: 1 2016-10-20 14:34:47,280 DEBUG [AsyncRpcChannel-pool2-t4] ipc.AsyncRpcChannel$8(566): IPC Client (-859809701) to cn012.l42scl.hortonworks.com/172.18.128.12:34094 from hbase.hfs.1: closed 2016-10-20 14:34:47,281 INFO [RS:0;cn012:34206] hbase.ChoreService(323): Chore service for: cn012.l42scl.hortonworks.com,34206,1476973982007 had [[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: cn012.l42scl.hortonworks.com,34206,1476973982007-MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region cn012.l42scl.hortonworks.com,34206,1476973982007 Period: 120000 Unit: MILLISECONDS]] on shutdown 2016-10-20 14:34:47,371 INFO [cn012.l42scl.hortonworks.com,34094,1476973981831_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2016-10-20 14:34:47,493 INFO [RS_CLOSE_REGION-cn012:34094-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=6, memsize=344, hasBloomFilter=true, into tmp file hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/namespace/37fa3cc8df36677363de1aacf6de184f/.tmp/7e0c8ab9a8404c5eada636adc33655d6 2016-10-20 14:34:47,495 INFO [RS_CLOSE_META-cn012:34094-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=15, memsize=3.4 K, hasBloomFilter=false, into tmp file hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/.tmp/6ddc1955e85b4a3aa68d31eb1c6d150e 2016-10-20 14:34:47,513 DEBUG [RS_CLOSE_REGION-cn012:34094-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/namespace/37fa3cc8df36677363de1aacf6de184f/.tmp/7e0c8ab9a8404c5eada636adc33655d6 as hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/namespace/37fa3cc8df36677363de1aacf6de184f/info/7e0c8ab9a8404c5eada636adc33655d6 2016-10-20 14:34:47,523 INFO [IPC Server handler 8 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-651c82c6-88ce-4f3b-bae6-e3460c66df10:NORMAL:127.0.0.1:43453|FINALIZED]]} size 0 2016-10-20 14:34:47,524 INFO [RS_CLOSE_META-cn012:34094-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=15, memsize=704, hasBloomFilter=false, into tmp file hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/.tmp/819a4e14691a4744a16025d50f5f8fbd 2016-10-20 14:34:47,526 INFO [RS_CLOSE_REGION-cn012:34094-0] regionserver.HStore(934): Added hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/namespace/37fa3cc8df36677363de1aacf6de184f/info/7e0c8ab9a8404c5eada636adc33655d6, entries=2, sequenceid=6, filesize=4.8 K 2016-10-20 14:34:47,528 INFO [RS_CLOSE_REGION-cn012:34094-0] regionserver.HRegion(2545): Finished memstore flush of ~344 B/344, currentsize=0 B/0 for region hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f. in 459ms, sequenceid=6, compaction requested=false 2016-10-20 14:34:47,533 INFO [StoreCloserThread-hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f.-1] regionserver.HStore(839): Closed info 2016-10-20 14:34:47,541 DEBUG [RS_CLOSE_META-cn012:34094-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/.tmp/6ddc1955e85b4a3aa68d31eb1c6d150e as hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/info/6ddc1955e85b4a3aa68d31eb1c6d150e 2016-10-20 14:34:47,542 DEBUG [RS_CLOSE_REGION-cn012:34094-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/namespace/37fa3cc8df36677363de1aacf6de184f/recovered.edits/9.seqid to file, newSeqId=9, maxSeqId=2 2016-10-20 14:34:47,544 INFO [RS_CLOSE_REGION-cn012:34094-0] regionserver.HRegion(1552): Closed hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f. 2016-10-20 14:34:47,544 DEBUG [RS_CLOSE_REGION-cn012:34094-0] handler.CloseRegionHandler(122): Closed hbase:namespace,,1476973984099.37fa3cc8df36677363de1aacf6de184f. 2016-10-20 14:34:47,553 INFO [RS_CLOSE_META-cn012:34094-0] regionserver.HStore(934): Added hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/info/6ddc1955e85b4a3aa68d31eb1c6d150e, entries=14, sequenceid=15, filesize=6.3 K 2016-10-20 14:34:47,555 DEBUG [RS_CLOSE_META-cn012:34094-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/.tmp/819a4e14691a4744a16025d50f5f8fbd as hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/table/819a4e14691a4744a16025d50f5f8fbd 2016-10-20 14:34:47,565 INFO [RS_CLOSE_META-cn012:34094-0] regionserver.HStore(934): Added hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/table/819a4e14691a4744a16025d50f5f8fbd, entries=4, sequenceid=15, filesize=4.7 K 2016-10-20 14:34:47,567 INFO [RS_CLOSE_META-cn012:34094-0] regionserver.HRegion(2545): Finished memstore flush of ~4.08 KB/4176, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 483ms, sequenceid=15, compaction requested=false 2016-10-20 14:34:47,572 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(839): Closed info 2016-10-20 14:34:47,577 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(839): Closed table 2016-10-20 14:34:47,585 DEBUG [RS_CLOSE_META-cn012:34094-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:46268/user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/data/hbase/meta/1588230740/recovered.edits/18.seqid to file, newSeqId=18, maxSeqId=3 2016-10-20 14:34:47,586 DEBUG [RS_CLOSE_META-cn012:34094-0] coprocessor.CoprocessorHost(271): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2016-10-20 14:34:47,586 INFO [RS_CLOSE_META-cn012:34094-0] regionserver.HRegion(1552): Closed hbase:meta,,1.1588230740 2016-10-20 14:34:47,587 DEBUG [RS_CLOSE_META-cn012:34094-0] handler.CloseRegionHandler(122): Closed hbase:meta,,1.1588230740 2016-10-20 14:34:47,681 INFO [M:0;cn012:34094] regionserver.HRegionServer(1091): stopping server cn012.l42scl.hortonworks.com,34094,1476973981831; all regions closed. 2016-10-20 14:34:47,682 DEBUG [M:0;cn012:34094] wal.FSHLog(1085): Closing WAL writer in /user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:34:47,688 INFO [IPC Server handler 3 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-651c82c6-88ce-4f3b-bae6-e3460c66df10:NORMAL:127.0.0.1:43453|RBW]]} size 83 2016-10-20 14:34:47,693 DEBUG [M:0;cn012:34094] wal.FSHLog(1043): Moved 1 WAL file(s) to /user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/oldWALs 2016-10-20 14:34:47,693 INFO [M:0;cn012:34094] wal.FSHLog(1046): Closed WAL: FSHLog cn012.l42scl.hortonworks.com%2C34094%2C1476973981831.meta:.meta(num 1476973983846) 2016-10-20 14:34:47,693 DEBUG [M:0;cn012:34094] wal.FSHLog(1085): Closing WAL writer in /user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:34:47,707 INFO [IPC Server handler 5 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-651c82c6-88ce-4f3b-bae6-e3460c66df10:NORMAL:127.0.0.1:43453|RBW]]} size 83 2016-10-20 14:34:47,712 DEBUG [M:0;cn012:34094] wal.FSHLog(1043): Moved 1 WAL file(s) to /user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/oldWALs 2016-10-20 14:34:47,712 INFO [M:0;cn012:34094] wal.FSHLog(1046): Closed WAL: FSHLog cn012.l42scl.hortonworks.com%2C34094%2C1476973981831.regiongroup-0:(num 1476973984847) 2016-10-20 14:34:47,712 DEBUG [M:0;cn012:34094] wal.FSHLog(1085): Closing WAL writer in /user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/WALs/cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:34:47,718 INFO [IPC Server handler 3 on 46268] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43453 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-a82af92a-d58e-4076-affe-976f93711253:NORMAL:127.0.0.1:43453|RBW]]} size 83 2016-10-20 14:34:47,722 DEBUG [M:0;cn012:34094] wal.FSHLog(1043): Moved 1 WAL file(s) to /user/hbase/test-data/cd059da3-acee-4d21-bf44-3f0ff38f8998/oldWALs 2016-10-20 14:34:47,722 INFO [M:0;cn012:34094] wal.FSHLog(1046): Closed WAL: FSHLog cn012.l42scl.hortonworks.com%2C34094%2C1476973981831.regiongroup-1:(num 1476973984907) 2016-10-20 14:34:47,722 DEBUG [M:0;cn012:34094] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:47,722 INFO [M:0;cn012:34094] regionserver.Leases(146): M:0;cn012:34094 closing leases 2016-10-20 14:34:47,722 INFO [M:0;cn012:34094] regionserver.Leases(149): M:0;cn012:34094 closed leases 2016-10-20 14:34:47,723 INFO [M:0;cn012:34094] hbase.ChoreService(323): Chore service for: cn012.l42scl.hortonworks.com,34094,1476973981831 had [[ScheduledChore: Name: cn012.l42scl.hortonworks.com,34094,1476973981831-MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: cn012.l42scl.hortonworks.com,34094,1476973981831-MobCompactionChore Period: 604800 Unit: SECONDS], [ScheduledChore: Name: cn012.l42scl.hortonworks.com,34094,1476973981831-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: cn012.l42scl.hortonworks.com,34094,1476973981831-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: cn012.l42scl.hortonworks.com,34094,1476973981831-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS], [ScheduledChore: Name: cn012.l42scl.hortonworks.com,34094,1476973981831-RegionNormalizerChore Period: 1800000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region cn012.l42scl.hortonworks.com,34094,1476973981831 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-cn012:34094 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS]] on shutdown 2016-10-20 14:34:47,767 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker] regionserver.Leases(146): regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker closing leases 2016-10-20 14:34:47,768 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker] regionserver.Leases(149): regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker closed leases 2016-10-20 14:34:47,791 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker] regionserver.Leases(146): master/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker closing leases 2016-10-20 14:34:47,791 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] regionserver.LogRoller(170): LogRoller exiting. 2016-10-20 14:34:47,791 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker] regionserver.Leases(149): master/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker closed leases 2016-10-20 14:34:47,803 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] regionserver.LogRoller(170): LogRoller exiting. 2016-10-20 14:34:47,803 INFO [RS:0;cn012:34206] regionserver.CompactSplitThread(403): Waiting for Split Thread to finish... 2016-10-20 14:34:47,804 INFO [RS:0;cn012:34206] regionserver.CompactSplitThread(403): Waiting for Merge Thread to finish... 2016-10-20 14:34:47,804 INFO [RS:0;cn012:34206] regionserver.CompactSplitThread(403): Waiting for Large Compaction Thread to finish... 2016-10-20 14:34:47,805 INFO [RS:0;cn012:34206] regionserver.CompactSplitThread(403): Waiting for Small Compaction Thread to finish... 2016-10-20 14:34:47,811 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:34206-0x157e2823ec60008, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/replication/rs/cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:34:47,811 INFO [RS:0;cn012:34206] ipc.RpcServer(2336): Stopping server on 34206 2016-10-20 14:34:47,814 INFO [RpcServer.listener,port=34206] ipc.RpcServer$Listener(816): RpcServer.listener,port=34206: stopping 2016-10-20 14:34:47,814 INFO [RpcServer.responder] ipc.RpcServer$Responder(1059): RpcServer.responder: stopped 2016-10-20 14:34:47,815 INFO [RpcServer.responder] ipc.RpcServer$Responder(962): RpcServer.responder: stopping 2016-10-20 14:34:47,845 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:34206-0x157e2823ec60008, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/rs/cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:34:47,845 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/rs/cn012.l42scl.hortonworks.com,34206,1476973982007 2016-10-20 14:34:47,846 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:34206-0x157e2823ec60008, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/2/rs 2016-10-20 14:34:47,846 INFO [main-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [cn012.l42scl.hortonworks.com,34206,1476973982007] 2016-10-20 14:34:47,856 INFO [RS_OPEN_META-cn012:34094-0-MetaLogRoller] regionserver.LogRoller(170): LogRoller exiting. 2016-10-20 14:34:47,862 INFO [main-EventThread] master.ServerManager(609): Cluster shutdown set; cn012.l42scl.hortonworks.com,34206,1476973982007 expired; onlineServers=1 2016-10-20 14:34:47,862 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/2/rs 2016-10-20 14:34:47,869 INFO [RS:0;cn012:34206] regionserver.HRegionServer(1135): stopping server cn012.l42scl.hortonworks.com,34206,1476973982007; zookeeper connection closed. 2016-10-20 14:34:47,869 INFO [RS:0;cn012:34206] regionserver.HRegionServer(1138): RS:0;cn012:34206 exiting 2016-10-20 14:34:47,870 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@8e8ecd2] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(190): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@8e8ecd2 2016-10-20 14:34:47,871 INFO [main] util.JVMClusterUtil(317): Shutdown of 1 master(s) and 1 regionserver(s) complete 2016-10-20 14:34:47,877 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/replication/rs/cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:34:47,878 INFO [M:0;cn012:34094] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2016-10-20 14:34:47,878 INFO [M:0;cn012:34094] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2016-10-20 14:34:47,878 DEBUG [M:0;cn012:34094] master.HMaster(1133): Stopping service threads 2016-10-20 14:34:47,885 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/master 2016-10-20 14:34:47,886 INFO [M:0;cn012:34094] hbase.ChoreService(323): Chore service for: cn012.l42scl.hortonworks.com,34094,1476973981831_splitLogManager_ had [] on shutdown 2016-10-20 14:34:47,886 INFO [M:0;cn012:34094] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2016-10-20 14:34:47,886 INFO [M:0;cn012:34094] master.LogRollMasterProcedureManager(55): stop: server shutting down. 2016-10-20 14:34:47,886 DEBUG [main-EventThread] zookeeper.ZKUtil(367): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Set watcher on znode that does not yet exist, /2/master 2016-10-20 14:34:47,887 INFO [M:0;cn012:34094] ipc.RpcServer(2336): Stopping server on 34094 2016-10-20 14:34:47,887 INFO [RpcServer.listener,port=34094] ipc.RpcServer$Listener(816): RpcServer.listener,port=34094: stopping 2016-10-20 14:34:47,889 INFO [RpcServer.responder] ipc.RpcServer$Responder(1059): RpcServer.responder: stopped 2016-10-20 14:34:47,890 INFO [RpcServer.responder] ipc.RpcServer$Responder(962): RpcServer.responder: stopping 2016-10-20 14:34:47,894 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:34094-0x157e2823ec60007, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/2/rs/cn012.l42scl.hortonworks.com,34094,1476973981831 2016-10-20 14:34:47,894 INFO [main-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [cn012.l42scl.hortonworks.com,34094,1476973981831] 2016-10-20 14:34:47,902 INFO [M:0;cn012:34094] regionserver.HRegionServer(1135): stopping server cn012.l42scl.hortonworks.com,34094,1476973981831; zookeeper connection closed. 2016-10-20 14:34:47,902 INFO [M:0;cn012:34094] regionserver.HRegionServer(1138): M:0;cn012:34094 exiting 2016-10-20 14:34:47,903 WARN [main] datanode.DirectoryScanner(378): DirectoryScanner: shutdown has been called 2016-10-20 14:34:47,924 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2016-10-20 14:34:48,029 WARN [DataNode: [[[DISK]file:/mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/dfscluster_0567527d-a565-4479-bfb0-c584662a6b9b/dfs/data/data1/, [DISK]file:/mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/dfscluster_0567527d-a565-4479-bfb0-c584662a6b9b/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:46268] datanode.BPServiceActor(704): BPOfferService for Block pool BP-1570554338-172.18.128.12-1476973980237 (Datanode Uuid fe6ec94c-4b95-4177-bb8b-a2034fdb7a43) service to localhost/127.0.0.1:46268 interrupted 2016-10-20 14:34:48,038 WARN [DataNode: [[[DISK]file:/mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/dfscluster_0567527d-a565-4479-bfb0-c584662a6b9b/dfs/data/data1/, [DISK]file:/mnt/disk2/a/7912/hbase-server/target/test-data/e953dd7b-a12a-43f5-9e3c-59c5ca6bec99/dfscluster_0567527d-a565-4479-bfb0-c584662a6b9b/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:46268] datanode.BPServiceActor(834): Ending block pool service for: Block pool BP-1570554338-172.18.128.12-1476973980237 (Datanode Uuid fe6ec94c-4b95-4177-bb8b-a2034fdb7a43) service to localhost/127.0.0.1:46268 2016-10-20 14:34:48,094 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2016-10-20 14:34:48,252 INFO [main] hbase.HBaseTestingUtility(1156): Minicluster is down 2016-10-20 14:34:48,252 INFO [main] hbase.HBaseTestingUtility(1143): Shutting down minicluster 2016-10-20 14:34:48,252 INFO [main] client.ConnectionImplementation(1805): Closing master protocol: MasterService 2016-10-20 14:34:48,252 INFO [main] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec60006 2016-10-20 14:34:48,261 DEBUG [main] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:48,261 DEBUG [main] util.JVMClusterUtil(241): Shutting down HBase Cluster 2016-10-20 14:34:48,261 DEBUG [main] coprocessor.CoprocessorHost(271): Stop coprocessor org.apache.hadoop.hbase.backup.master.BackupController 2016-10-20 14:34:48,261 INFO [main] regionserver.HRegionServer(1923): STOPPED: Cluster shutdown requested 2016-10-20 14:34:48,261 DEBUG [RpcServer.reader=0,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42532 because read count=-1. Number of active connections: 7 2016-10-20 14:34:48,261 INFO [M:0;cn012:41289] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2016-10-20 14:34:48,262 INFO [M:0;cn012:41289] regionserver.HeapMemoryManager(202): Stoping HeapMemoryTuner chore. 2016-10-20 14:34:48,262 INFO [SplitLogWorker-cn012:41289] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2016-10-20 14:34:48,262 INFO [M:0;cn012:41289] procedure2.ProcedureExecutor(532): Stopping the procedure executor 2016-10-20 14:34:48,262 INFO [M:0;cn012:41289] wal.WALProcedureStore(232): Stopping the WAL Procedure Store 2016-10-20 14:34:48,261 DEBUG [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Listener(912): RpcServer.listener,port=37560: DISCONNECTING client 172.18.128.12:42062 because read count=-1. Number of active connections: 6 2016-10-20 14:34:48,261 DEBUG [AsyncRpcChannel-pool2-t15] ipc.AsyncRpcChannel$8(566): IPC Client (404076031) to cn012.l42scl.hortonworks.com/172.18.128.12:37560 from hbase: closed 2016-10-20 14:34:48,261 DEBUG [AsyncRpcChannel-pool2-t2] ipc.AsyncRpcChannel$8(566): IPC Client (-45329822) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:48,261 DEBUG [RpcServer.reader=2,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42401 because read count=-1. Number of active connections: 7 2016-10-20 14:34:48,261 DEBUG [AsyncRpcChannel-pool2-t7] ipc.AsyncRpcChannel$8(566): IPC Client (-127742945) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase: closed 2016-10-20 14:34:48,265 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2016-10-20 14:34:48,262 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2016-10-20 14:34:48,262 INFO [SplitLogWorker-cn012:41289] regionserver.SplitLogWorker(155): SplitLogWorker cn012.l42scl.hortonworks.com,41289,1476973974467 exiting 2016-10-20 14:34:48,271 INFO [main] regionserver.HRegionServer(1923): STOPPED: Shutdown requested 2016-10-20 14:34:48,271 INFO [RS:0;cn012:37560] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2016-10-20 14:34:48,271 INFO [RS:0;cn012:37560] regionserver.HeapMemoryManager(202): Stoping HeapMemoryTuner chore. 2016-10-20 14:34:48,271 INFO [SplitLogWorker-cn012:37560] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2016-10-20 14:34:48,272 INFO [RS:0;cn012:37560] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2016-10-20 14:34:48,272 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/running 2016-10-20 14:34:48,272 INFO [RS:0;cn012:37560] regionserver.LogRollRegionServerProcedureManager(96): Stopping RegionServerBackupManager gracefully. 2016-10-20 14:34:48,272 INFO [SplitLogWorker-cn012:37560] regionserver.SplitLogWorker(155): SplitLogWorker cn012.l42scl.hortonworks.com,37560,1476973974890 exiting 2016-10-20 14:34:48,276 INFO [RS:0;cn012:37560] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2016-10-20 14:34:48,276 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/running 2016-10-20 14:34:48,276 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2016-10-20 14:34:48,272 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2016-10-20 14:34:48,277 INFO [IPC Server handler 5 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741830_1006{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 68446 2016-10-20 14:34:48,277 INFO [RS:0;cn012:37560] regionserver.HRegionServer(1063): stopping server cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:48,278 DEBUG [RS:0;cn012:37560] zookeeper.MetaTableLocator(612): Stopping MetaTableLocator 2016-10-20 14:34:48,277 DEBUG [main-EventThread] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/running 2016-10-20 14:34:48,278 INFO [RS:0;cn012:37560] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec60002 2016-10-20 14:34:48,278 DEBUG [RS_CLOSE_REGION-cn012:37560-0] handler.CloseRegionHandler(90): Processing close of ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:34:48,277 DEBUG [RS_CLOSE_REGION-cn012:37560-1] handler.CloseRegionHandler(90): Processing close of ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e. 2016-10-20 14:34:48,277 DEBUG [RS_CLOSE_REGION-cn012:37560-2] handler.CloseRegionHandler(90): Processing close of ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:48,277 DEBUG [main-EventThread] zookeeper.ZKUtil(367): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/running 2016-10-20 14:34:48,281 DEBUG [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(1419): Closing ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e.: disabling compactions & flushes 2016-10-20 14:34:48,280 DEBUG [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1419): Closing ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.: disabling compactions & flushes 2016-10-20 14:34:48,279 DEBUG [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(1419): Closing ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.: disabling compactions & flushes 2016-10-20 14:34:48,281 DEBUG [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1446): Updates disabled for region ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:48,281 DEBUG [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(1446): Updates disabled for region ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e. 2016-10-20 14:34:48,281 DEBUG [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(1446): Updates disabled for region ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:34:48,282 INFO [StoreCloserThread-ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.-1] regionserver.HStore(839): Closed f 2016-10-20 14:34:48,282 INFO [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(2345): Flushing 2/2 column families, memstore=1.31 KB 2016-10-20 14:34:48,282 INFO [StoreCloserThread-ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e.-1] regionserver.HStore(839): Closed f 2016-10-20 14:34:48,290 INFO [StoreCloserThread-ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99.-1] regionserver.HStore(839): Closed f2 2016-10-20 14:34:48,290 DEBUG [RS:0;cn012:37560] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:48,290 INFO [RS:0;cn012:37560] regionserver.HRegionServer(1292): Waiting on 8 regions to close 2016-10-20 14:34:48,290 DEBUG [RS:0;cn012:37560] regionserver.HRegionServer(1296): {0bcceb715ec0f6a2bed0aef31b8a540e=ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e., 4a7f646a7c8fc84833ef454ba337a246=ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246., b35704e7c3c2eb86497b08776362fa99=ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99., 9d78a3921505ca9bb53c61810d6ae099=ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099., 81900e9262bb3c77f4b3fd7b6387aeb8=ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8., a1135d09b318d77673368ed569f0b066=ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066., 20fb71680e16fdda348e8695b59a8391=ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391., 7b31f6434ce41b4dbce476e8d5a763e0=hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0.} 2016-10-20 14:34:48,291 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:42622 because read count=-1. Number of active connections: 5 2016-10-20 14:34:48,291 DEBUG [AsyncRpcChannel-pool2-t13] ipc.AsyncRpcChannel$8(566): IPC Client (-1205399475) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase.hfs.0: closed 2016-10-20 14:34:48,299 DEBUG [RS_CLOSE_REGION-cn012:37560-2] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/b35704e7c3c2eb86497b08776362fa99/recovered.edits/11.seqid to file, newSeqId=11, maxSeqId=7 2016-10-20 14:34:48,299 DEBUG [RS_CLOSE_REGION-cn012:37560-1] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns3/test-14769740038902/0bcceb715ec0f6a2bed0aef31b8a540e/recovered.edits/4.seqid to file, newSeqId=4, maxSeqId=2 2016-10-20 14:34:48,299 INFO [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1552): Closed ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:48,300 DEBUG [RS_CLOSE_REGION-cn012:37560-2] handler.CloseRegionHandler(122): Closed ns1:table1_restore,,1476974046919.b35704e7c3c2eb86497b08776362fa99. 2016-10-20 14:34:48,300 DEBUG [RS_CLOSE_REGION-cn012:37560-2] handler.CloseRegionHandler(90): Processing close of ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099. 2016-10-20 14:34:48,302 DEBUG [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1419): Closing ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099.: disabling compactions & flushes 2016-10-20 14:34:48,302 DEBUG [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1446): Updates disabled for region ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099. 2016-10-20 14:34:48,302 INFO [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(1552): Closed ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e. 2016-10-20 14:34:48,302 INFO [StoreCloserThread-ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099.-1] regionserver.HStore(839): Closed f 2016-10-20 14:34:48,302 DEBUG [RS_CLOSE_REGION-cn012:37560-1] handler.CloseRegionHandler(122): Closed ns3:test-14769740038902,,1476974013079.0bcceb715ec0f6a2bed0aef31b8a540e. 2016-10-20 14:34:48,302 DEBUG [RS_CLOSE_REGION-cn012:37560-1] handler.CloseRegionHandler(90): Processing close of ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:48,303 DEBUG [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(1419): Closing ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8.: disabling compactions & flushes 2016-10-20 14:34:48,303 DEBUG [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(1446): Updates disabled for region ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:48,305 INFO [StoreCloserThread-ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8.-1] regionserver.HStore(839): Closed f 2016-10-20 14:34:48,306 INFO [IPC Server handler 6 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741929_1105{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 5358 2016-10-20 14:34:48,310 DEBUG [RS_CLOSE_REGION-cn012:37560-2] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns4/test-14769740038903/9d78a3921505ca9bb53c61810d6ae099/recovered.edits/4.seqid to file, newSeqId=4, maxSeqId=2 2016-10-20 14:34:48,311 INFO [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1552): Closed ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099. 2016-10-20 14:34:48,311 DEBUG [RS_CLOSE_REGION-cn012:37560-2] handler.CloseRegionHandler(122): Closed ns4:test-14769740038903,,1476974015387.9d78a3921505ca9bb53c61810d6ae099. 2016-10-20 14:34:48,311 DEBUG [RS_CLOSE_REGION-cn012:37560-2] handler.CloseRegionHandler(90): Processing close of ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:48,313 DEBUG [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1419): Closing ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.: disabling compactions & flushes 2016-10-20 14:34:48,313 DEBUG [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1446): Updates disabled for region ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:48,313 DEBUG [RS_CLOSE_REGION-cn012:37560-1] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/table2_restore/81900e9262bb3c77f4b3fd7b6387aeb8/recovered.edits/6.seqid to file, newSeqId=6, maxSeqId=2 2016-10-20 14:34:48,314 INFO [StoreCloserThread-ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.-1] regionserver.HStore(839): Closed f 2016-10-20 14:34:48,314 INFO [StoreCloserThread-ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066.-1] regionserver.HStore(839): Closed f2 2016-10-20 14:34:48,314 INFO [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(1552): Closed ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:48,315 DEBUG [RS_CLOSE_REGION-cn012:37560-1] handler.CloseRegionHandler(122): Closed ns2:table2_restore,,1476974049947.81900e9262bb3c77f4b3fd7b6387aeb8. 2016-10-20 14:34:48,315 DEBUG [RS_CLOSE_REGION-cn012:37560-1] handler.CloseRegionHandler(90): Processing close of ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. 2016-10-20 14:34:48,315 DEBUG [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(1419): Closing ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391.: disabling compactions & flushes 2016-10-20 14:34:48,315 DEBUG [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(1446): Updates disabled for region ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. 2016-10-20 14:34:48,315 INFO [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(2345): Flushing 1/1 column families, memstore=840 B 2016-10-20 14:34:48,319 DEBUG [RS_CLOSE_REGION-cn012:37560-2] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/table1_restore/a1135d09b318d77673368ed569f0b066/recovered.edits/9.seqid to file, newSeqId=9, maxSeqId=7 2016-10-20 14:34:48,320 INFO [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1552): Closed ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:48,320 DEBUG [RS_CLOSE_REGION-cn012:37560-2] handler.CloseRegionHandler(122): Closed ns1:table1_restore,row0,1476974046919.a1135d09b318d77673368ed569f0b066. 2016-10-20 14:34:48,320 DEBUG [RS_CLOSE_REGION-cn012:37560-2] handler.CloseRegionHandler(90): Processing close of hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0. 2016-10-20 14:34:48,322 DEBUG [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1419): Closing hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0.: disabling compactions & flushes 2016-10-20 14:34:48,322 DEBUG [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1446): Updates disabled for region hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0. 2016-10-20 14:34:48,322 INFO [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(2345): Flushing 2/2 column families, memstore=20.70 KB 2016-10-20 14:34:48,326 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741930_1106{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 5011 2016-10-20 14:34:48,332 INFO [IPC Server handler 5 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741931_1107{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 13133 2016-10-20 14:34:48,617 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@3dddefd8] blockmanagement.BlockManager(3488): BLOCK* BlockManager: ask 127.0.0.1:38642 to delete [blk_1073741856_1032, blk_1073741907_1083, blk_1073741908_1084, blk_1073741911_1087, blk_1073741919_1095] 2016-10-20 14:34:48,679 INFO [M:0;cn012:41289] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2016-10-20 14:34:48,679 INFO [M:0;cn012:41289] regionserver.LogRollRegionServerProcedureManager(96): Stopping RegionServerBackupManager gracefully. 2016-10-20 14:34:48,679 INFO [M:0;cn012:41289] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2016-10-20 14:34:48,680 INFO [M:0;cn012:41289] regionserver.HRegionServer(1063): stopping server cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:48,680 DEBUG [M:0;cn012:41289] zookeeper.MetaTableLocator(612): Stopping MetaTableLocator 2016-10-20 14:34:48,680 DEBUG [RS_CLOSE_REGION-cn012:41289-0] handler.CloseRegionHandler(90): Processing close of hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962. 2016-10-20 14:34:48,680 INFO [M:0;cn012:41289] client.ConnectionImplementation(1346): Closing zookeeper sessionid=0x157e2823ec60003 2016-10-20 14:34:48,683 DEBUG [RS_CLOSE_REGION-cn012:41289-0] regionserver.HRegion(1419): Closing hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962.: disabling compactions & flushes 2016-10-20 14:34:48,683 DEBUG [RS_CLOSE_REGION-cn012:41289-0] regionserver.HRegion(1446): Updates disabled for region hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962. 2016-10-20 14:34:48,683 INFO [RS_CLOSE_REGION-cn012:41289-0] regionserver.HRegion(2345): Flushing 1/1 column families, memstore=1016 B 2016-10-20 14:34:48,687 DEBUG [M:0;cn012:41289] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:48,689 INFO [M:0;cn012:41289] regionserver.CompactSplitThread(403): Waiting for Split Thread to finish... 2016-10-20 14:34:48,689 INFO [M:0;cn012:41289] regionserver.CompactSplitThread(403): Waiting for Merge Thread to finish... 2016-10-20 14:34:48,689 DEBUG [AsyncRpcChannel-pool2-t3] ipc.AsyncRpcChannel$8(566): IPC Client (-150393018) to cn012.l42scl.hortonworks.com/172.18.128.12:37560 from hbase: closed 2016-10-20 14:34:48,689 INFO [M:0;cn012:41289] regionserver.CompactSplitThread(403): Waiting for Large Compaction Thread to finish... 2016-10-20 14:34:48,690 INFO [M:0;cn012:41289] regionserver.CompactSplitThread(403): Waiting for Small Compaction Thread to finish... 2016-10-20 14:34:48,689 DEBUG [AsyncRpcChannel-pool2-t14] ipc.AsyncRpcChannel$8(566): IPC Client (-614186705) to cn012.l42scl.hortonworks.com/172.18.128.12:37560 from hbase: closed 2016-10-20 14:34:48,689 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Listener(912): RpcServer.listener,port=37560: DISCONNECTING client 172.18.128.12:41784 because read count=-1. Number of active connections: 5 2016-10-20 14:34:48,690 INFO [M:0;cn012:41289] regionserver.HRegionServer(1292): Waiting on 2 regions to close 2016-10-20 14:34:48,691 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=37560] ipc.RpcServer$Listener(912): RpcServer.listener,port=37560: DISCONNECTING client 172.18.128.12:41992 because read count=-1. Number of active connections: 4 2016-10-20 14:34:48,691 DEBUG [RS_CLOSE_META-cn012:41289-0] handler.CloseRegionHandler(90): Processing close of hbase:meta,,1.1588230740 2016-10-20 14:34:48,691 DEBUG [M:0;cn012:41289] regionserver.HRegionServer(1296): {b386968767678fdb930a9a40fbb05962=hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962., 1588230740=hbase:meta,,1.1588230740} 2016-10-20 14:34:48,700 DEBUG [RS_CLOSE_META-cn012:41289-0] regionserver.HRegion(1419): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2016-10-20 14:34:48,701 DEBUG [RS_CLOSE_META-cn012:41289-0] regionserver.HRegion(1446): Updates disabled for region hbase:meta,,1.1588230740 2016-10-20 14:34:48,701 INFO [RS_CLOSE_META-cn012:41289-0] regionserver.HRegion(2345): Flushing 2/2 column families, memstore=32.90 KB 2016-10-20 14:34:48,708 INFO [RS_CLOSE_REGION-cn012:37560-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=230, memsize=1.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/.tmp/c13b39cca154403a901f98d4f8933403 2016-10-20 14:34:48,716 INFO [IPC Server handler 3 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741932_1108{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 5048 2016-10-20 14:34:48,720 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] regionserver.LogRoller(170): LogRoller exiting. 2016-10-20 14:34:48,721 INFO [IPC Server handler 2 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741933_1109{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|FINALIZED]]} size 0 2016-10-20 14:34:48,722 INFO [RS_CLOSE_META-cn012:41289-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=81, memsize=29.7 K, hasBloomFilter=false, into tmp file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/.tmp/9ad9fbc10eb24875ace70bd78a5f29ab 2016-10-20 14:34:48,722 DEBUG [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/.tmp/c13b39cca154403a901f98d4f8933403 as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f2/c13b39cca154403a901f98d4f8933403 2016-10-20 14:34:48,726 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.logRoller] regionserver.LogRoller(170): LogRoller exiting. 2016-10-20 14:34:48,727 INFO [RS_CLOSE_REGION-cn012:37560-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=111, memsize=840, hasBloomFilter=true, into tmp file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/.tmp/0e09a21950b84acebe781e3fe56c38a6 2016-10-20 14:34:48,732 INFO [RS_CLOSE_META-cn012:41289-0] regionserver.StoreFile$Reader(1606): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9ad9fbc10eb24875ace70bd78a5f29ab 2016-10-20 14:34:48,733 INFO [RS_CLOSE_REGION-cn012:37560-0] regionserver.HStore(934): Added hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/f2/c13b39cca154403a901f98d4f8933403, entries=7, sequenceid=230, filesize=5.2 K 2016-10-20 14:34:48,733 INFO [RS_CLOSE_REGION-cn012:37560-2] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=30, memsize=16.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/backup/7b31f6434ce41b4dbce476e8d5a763e0/.tmp/079b0c825d654898a36025e76f9f4e09 2016-10-20 14:34:48,735 INFO [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(2545): Finished memstore flush of ~1.31 KB/1344, currentsize=0 B/0 for region ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. in 453ms, sequenceid=230, compaction requested=false 2016-10-20 14:34:48,740 INFO [StoreCloserThread-ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.-1] regionserver.HStore(839): Closed f 2016-10-20 14:34:48,744 INFO [StoreCloserThread-ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246.-1] regionserver.HStore(839): Closed f2 2016-10-20 14:34:48,745 DEBUG [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/.tmp/0e09a21950b84acebe781e3fe56c38a6 as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/f/0e09a21950b84acebe781e3fe56c38a6 2016-10-20 14:34:48,749 INFO [IPC Server handler 5 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741934_1110{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 5583 2016-10-20 14:34:48,750 DEBUG [RS_CLOSE_REGION-cn012:37560-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns1/test-1476974003890/4a7f646a7c8fc84833ef454ba337a246/recovered.edits/233.seqid to file, newSeqId=233, maxSeqId=221 2016-10-20 14:34:48,752 INFO [RS_CLOSE_REGION-cn012:37560-0] regionserver.HRegion(1552): Closed ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:34:48,752 DEBUG [RS_CLOSE_REGION-cn012:37560-0] handler.CloseRegionHandler(122): Closed ns1:test-1476974003890,,1476974009047.4a7f646a7c8fc84833ef454ba337a246. 2016-10-20 14:34:48,755 INFO [RS_CLOSE_REGION-cn012:37560-1] regionserver.HStore(934): Added hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/f/0e09a21950b84acebe781e3fe56c38a6, entries=5, sequenceid=111, filesize=4.9 K 2016-10-20 14:34:48,755 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467_ChoreService_1] hbase.ScheduledChore(179): Chore: cn012.l42scl.hortonworks.com,41289,1476973974467-MemstoreFlusherChore was stopped 2016-10-20 14:34:48,756 INFO [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(2545): Finished memstore flush of ~840 B/840, currentsize=0 B/0 for region ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. in 441ms, sequenceid=111, compaction requested=false 2016-10-20 14:34:48,766 INFO [StoreCloserThread-ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391.-1] regionserver.HStore(839): Closed f 2016-10-20 14:34:48,769 INFO [IPC Server handler 1 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741935_1111{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 6158 2016-10-20 14:34:48,772 DEBUG [RS_CLOSE_REGION-cn012:37560-1] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/ns2/test-14769740038901/20fb71680e16fdda348e8695b59a8391/recovered.edits/114.seqid to file, newSeqId=114, maxSeqId=2 2016-10-20 14:34:48,773 INFO [RS_CLOSE_REGION-cn012:37560-1] regionserver.HRegion(1552): Closed ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. 2016-10-20 14:34:48,773 DEBUG [RS_CLOSE_REGION-cn012:37560-1] handler.CloseRegionHandler(122): Closed ns2:test-14769740038901,,1476974011603.20fb71680e16fdda348e8695b59a8391. 2016-10-20 14:34:48,803 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker] regionserver.Leases(146): master/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker closing leases 2016-10-20 14:34:48,803 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker] regionserver.Leases(146): regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker closing leases 2016-10-20 14:34:48,803 INFO [master/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker] regionserver.Leases(149): master/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker closed leases 2016-10-20 14:34:48,803 INFO [regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker] regionserver.Leases(149): regionserver/cn012.l42scl.hortonworks.com/172.18.128.12:0.leaseChecker closed leases 2016-10-20 14:34:48,828 INFO [cn012.l42scl.hortonworks.com,37560,1476973974890_ChoreService_1] hbase.ScheduledChore(179): Chore: cn012.l42scl.hortonworks.com,37560,1476973974890-MemstoreFlusherChore was stopped 2016-10-20 14:34:48,922 INFO [RS_OPEN_META-cn012:41289-0-MetaLogRoller] regionserver.LogRoller(170): LogRoller exiting. 2016-10-20 14:34:49,118 INFO [RS_CLOSE_REGION-cn012:41289-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=10, memsize=1016, hasBloomFilter=true, into tmp file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/namespace/b386968767678fdb930a9a40fbb05962/.tmp/9ee287b03faa48e1b009853a3900b148 2016-10-20 14:34:49,138 DEBUG [RS_CLOSE_REGION-cn012:41289-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/namespace/b386968767678fdb930a9a40fbb05962/.tmp/9ee287b03faa48e1b009853a3900b148 as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/namespace/b386968767678fdb930a9a40fbb05962/info/9ee287b03faa48e1b009853a3900b148 2016-10-20 14:34:49,151 INFO [RS_CLOSE_REGION-cn012:41289-0] regionserver.HStore(934): Added hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/namespace/b386968767678fdb930a9a40fbb05962/info/9ee287b03faa48e1b009853a3900b148, entries=6, sequenceid=10, filesize=4.9 K 2016-10-20 14:34:49,151 INFO [RS_CLOSE_META-cn012:41289-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=81, memsize=3.2 K, hasBloomFilter=false, into tmp file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/.tmp/bd12744b05b844bb8920d1fb8f16af0c 2016-10-20 14:34:49,153 INFO [RS_CLOSE_REGION-cn012:41289-0] regionserver.HRegion(2545): Finished memstore flush of ~1016 B/1016, currentsize=0 B/0 for region hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962. in 470ms, sequenceid=10, compaction requested=false 2016-10-20 14:34:49,158 INFO [StoreCloserThread-hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962.-1] regionserver.HStore(839): Closed info 2016-10-20 14:34:49,167 DEBUG [RS_CLOSE_REGION-cn012:41289-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/namespace/b386968767678fdb930a9a40fbb05962/recovered.edits/13.seqid to file, newSeqId=13, maxSeqId=2 2016-10-20 14:34:49,169 INFO [RS_CLOSE_REGION-cn012:41289-0] regionserver.HRegion(1552): Closed hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962. 2016-10-20 14:34:49,169 DEBUG [RS_CLOSE_REGION-cn012:41289-0] handler.CloseRegionHandler(122): Closed hbase:namespace,,1476973976394.b386968767678fdb930a9a40fbb05962. 2016-10-20 14:34:49,169 DEBUG [RS_CLOSE_META-cn012:41289-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/.tmp/9ad9fbc10eb24875ace70bd78a5f29ab as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/info/9ad9fbc10eb24875ace70bd78a5f29ab 2016-10-20 14:34:49,170 INFO [RS_CLOSE_REGION-cn012:37560-2] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=30, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/backup/7b31f6434ce41b4dbce476e8d5a763e0/.tmp/d7a8eb3f7ec6406aac156db26406db5f 2016-10-20 14:34:49,180 INFO [RS_CLOSE_META-cn012:41289-0] regionserver.StoreFile$Reader(1606): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9ad9fbc10eb24875ace70bd78a5f29ab 2016-10-20 14:34:49,181 INFO [RS_CLOSE_META-cn012:41289-0] regionserver.HStore(934): Added hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/info/9ad9fbc10eb24875ace70bd78a5f29ab, entries=114, sequenceid=81, filesize=18.6 K 2016-10-20 14:34:49,181 DEBUG [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/backup/7b31f6434ce41b4dbce476e8d5a763e0/.tmp/079b0c825d654898a36025e76f9f4e09 as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/backup/7b31f6434ce41b4dbce476e8d5a763e0/meta/079b0c825d654898a36025e76f9f4e09 2016-10-20 14:34:49,183 DEBUG [RS_CLOSE_META-cn012:41289-0] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/.tmp/bd12744b05b844bb8920d1fb8f16af0c as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/table/bd12744b05b844bb8920d1fb8f16af0c 2016-10-20 14:34:49,185 INFO [cn012.l42scl.hortonworks.com,41289,1476973974467_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2016-10-20 14:34:49,190 INFO [RS_CLOSE_REGION-cn012:37560-2] regionserver.HStore(934): Added hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/backup/7b31f6434ce41b4dbce476e8d5a763e0/meta/079b0c825d654898a36025e76f9f4e09, entries=43, sequenceid=30, filesize=12.8 K 2016-10-20 14:34:49,192 DEBUG [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegionFileSystem(382): Committing store file hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/backup/7b31f6434ce41b4dbce476e8d5a763e0/.tmp/d7a8eb3f7ec6406aac156db26406db5f as hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/backup/7b31f6434ce41b4dbce476e8d5a763e0/session/d7a8eb3f7ec6406aac156db26406db5f 2016-10-20 14:34:49,193 INFO [RS_CLOSE_META-cn012:41289-0] regionserver.HStore(934): Added hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/table/bd12744b05b844bb8920d1fb8f16af0c, entries=18, sequenceid=81, filesize=5.5 K 2016-10-20 14:34:49,195 INFO [RS_CLOSE_META-cn012:41289-0] regionserver.HRegion(2545): Finished memstore flush of ~32.90 KB/33688, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 494ms, sequenceid=81, compaction requested=false 2016-10-20 14:34:49,199 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(839): Closed info 2016-10-20 14:34:49,203 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(839): Closed table 2016-10-20 14:34:49,206 INFO [RS_CLOSE_REGION-cn012:37560-2] regionserver.HStore(934): Added hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/backup/7b31f6434ce41b4dbce476e8d5a763e0/session/d7a8eb3f7ec6406aac156db26406db5f, entries=3, sequenceid=30, filesize=6.0 K 2016-10-20 14:34:49,207 INFO [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(2545): Finished memstore flush of ~20.70 KB/21200, currentsize=0 B/0 for region hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0. in 885ms, sequenceid=30, compaction requested=false 2016-10-20 14:34:49,211 INFO [StoreCloserThread-hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0.-1] regionserver.HStore(839): Closed meta 2016-10-20 14:34:49,214 INFO [StoreCloserThread-hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0.-1] regionserver.HStore(839): Closed session 2016-10-20 14:34:49,216 DEBUG [RS_CLOSE_META-cn012:41289-0] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/meta/1588230740/recovered.edits/84.seqid to file, newSeqId=84, maxSeqId=3 2016-10-20 14:34:49,217 DEBUG [RS_CLOSE_META-cn012:41289-0] coprocessor.CoprocessorHost(271): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2016-10-20 14:34:49,217 INFO [RS_CLOSE_META-cn012:41289-0] regionserver.HRegion(1552): Closed hbase:meta,,1.1588230740 2016-10-20 14:34:49,217 DEBUG [RS_CLOSE_META-cn012:41289-0] handler.CloseRegionHandler(122): Closed hbase:meta,,1.1588230740 2016-10-20 14:34:49,219 DEBUG [RS_CLOSE_REGION-cn012:37560-2] wal.WALSplitter(730): Wrote region seqId=hdfs://localhost:37879/user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/data/hbase/backup/7b31f6434ce41b4dbce476e8d5a763e0/recovered.edits/33.seqid to file, newSeqId=33, maxSeqId=2 2016-10-20 14:34:49,220 INFO [RS_CLOSE_REGION-cn012:37560-2] regionserver.HRegion(1552): Closed hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0. 2016-10-20 14:34:49,220 DEBUG [RS_CLOSE_REGION-cn012:37560-2] handler.CloseRegionHandler(122): Closed hbase:backup,,1476973980098.7b31f6434ce41b4dbce476e8d5a763e0. 2016-10-20 14:34:49,291 INFO [RS:0;cn012:37560] regionserver.HRegionServer(1091): stopping server cn012.l42scl.hortonworks.com,37560,1476973974890; all regions closed. 2016-10-20 14:34:49,291 DEBUG [RS:0;cn012:37560] wal.FSHLog(1085): Closing WAL writer in /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:49,296 INFO [IPC Server handler 0 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741890_1066{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 5343 2016-10-20 14:34:49,301 INFO [M:0;cn012:41289] regionserver.HRegionServer(1091): stopping server cn012.l42scl.hortonworks.com,41289,1476973974467; all regions closed. 2016-10-20 14:34:49,301 DEBUG [M:0;cn012:41289] wal.FSHLog(1085): Closing WAL writer in /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:49,305 INFO [IPC Server handler 9 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 83 2016-10-20 14:34:49,309 DEBUG [M:0;cn012:41289] wal.FSHLog(1043): Moved 1 WAL file(s) to /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs 2016-10-20 14:34:49,309 INFO [M:0;cn012:41289] wal.FSHLog(1046): Closed WAL: FSHLog cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.meta:.meta(num 1476973975915) 2016-10-20 14:34:49,310 DEBUG [M:0;cn012:41289] wal.FSHLog(1085): Closing WAL writer in /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:49,315 INFO [IPC Server handler 6 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741889_1065{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 83 2016-10-20 14:34:49,319 DEBUG [M:0;cn012:41289] wal.FSHLog(1043): Moved 1 WAL file(s) to /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs 2016-10-20 14:34:49,319 INFO [M:0;cn012:41289] wal.FSHLog(1046): Closed WAL: FSHLog cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-0:(num 1476974042856) 2016-10-20 14:34:49,319 DEBUG [M:0;cn012:41289] wal.FSHLog(1085): Closing WAL writer in /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:49,323 INFO [IPC Server handler 9 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741892_1068{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ed22b33-45b0-4f46-8038-08522ec2da53:NORMAL:127.0.0.1:38642|RBW]]} size 83 2016-10-20 14:34:49,328 DEBUG [M:0;cn012:41289] wal.FSHLog(1043): Moved 2 WAL file(s) to /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs 2016-10-20 14:34:49,329 INFO [M:0;cn012:41289] wal.FSHLog(1046): Closed WAL: FSHLog cn012.l42scl.hortonworks.com%2C41289%2C1476973974467.regiongroup-1:(num 1476974043284) 2016-10-20 14:34:49,329 DEBUG [M:0;cn012:41289] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:49,329 INFO [M:0;cn012:41289] regionserver.Leases(146): M:0;cn012:41289 closing leases 2016-10-20 14:34:49,329 INFO [M:0;cn012:41289] regionserver.Leases(149): M:0;cn012:41289 closed leases 2016-10-20 14:34:49,329 INFO [M:0;cn012:41289] hbase.ChoreService(323): Chore service for: cn012.l42scl.hortonworks.com,41289,1476973974467 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-cn012:41289 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: cn012.l42scl.hortonworks.com,41289,1476973974467-MobCompactionChore Period: 604800 Unit: SECONDS], [ScheduledChore: Name: cn012.l42scl.hortonworks.com,41289,1476973974467-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: cn012.l42scl.hortonworks.com,41289,1476973974467-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: cn012.l42scl.hortonworks.com,41289,1476973974467-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: cn012.l42scl.hortonworks.com,41289,1476973974467-RegionNormalizerChore Period: 1800000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region cn012.l42scl.hortonworks.com,41289,1476973974467 Period: 120000 Unit: MILLISECONDS]] on shutdown 2016-10-20 14:34:49,335 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/replication/rs/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:49,335 INFO [M:0;cn012:41289] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2016-10-20 14:34:49,335 INFO [M:0;cn012:41289] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2016-10-20 14:34:49,336 INFO [M:0;cn012:41289] master.ServerManager(554): Waiting on regionserver(s) to go down cn012.l42scl.hortonworks.com,37560,1476973974890, cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:49,705 DEBUG [RS:0;cn012:37560] wal.FSHLog(1043): Moved 2 WAL file(s) to /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs 2016-10-20 14:34:49,706 INFO [RS:0;cn012:37560] wal.FSHLog(1046): Closed WAL: FSHLog cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-0:(num 1476974042857) 2016-10-20 14:34:49,706 DEBUG [RS:0;cn012:37560] wal.FSHLog(1085): Closing WAL writer in /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/WALs/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:49,712 INFO [IPC Server handler 8 on 37879] blockmanagement.BlockManager(2624): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38642 is added to blk_1073741891_1067{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c31187f8-167e-4840-a137-2273c32903cb:NORMAL:127.0.0.1:38642|RBW]]} size 9032 2016-10-20 14:34:50,166 DEBUG [RS:0;cn012:37560] wal.FSHLog(1043): Moved 4 WAL file(s) to /user/hbase/test-data/c6bcd31f-ff0b-4117-a391-b69050a0882e/oldWALs 2016-10-20 14:34:50,166 INFO [RS:0;cn012:37560] wal.FSHLog(1046): Closed WAL: FSHLog cn012.l42scl.hortonworks.com%2C37560%2C1476973974890.regiongroup-1:(num 1476974043279) 2016-10-20 14:34:50,167 DEBUG [RS:0;cn012:37560] ipc.AsyncRpcClient(320): Stopping async HBase RPC client 2016-10-20 14:34:50,167 INFO [RS:0;cn012:37560] regionserver.Leases(146): RS:0;cn012:37560 closing leases 2016-10-20 14:34:50,167 INFO [RS:0;cn012:37560] regionserver.Leases(149): RS:0;cn012:37560 closed leases 2016-10-20 14:34:50,167 DEBUG [RpcServer.reader=1,bindAddress=cn012.l42scl.hortonworks.com,port=41289] ipc.RpcServer$Listener(912): RpcServer.listener,port=41289: DISCONNECTING client 172.18.128.12:50334 because read count=-1. Number of active connections: 4 2016-10-20 14:34:50,167 DEBUG [AsyncRpcChannel-pool2-t1] ipc.AsyncRpcChannel$8(566): IPC Client (-459393209) to cn012.l42scl.hortonworks.com/172.18.128.12:41289 from hbase.hfs.0: closed 2016-10-20 14:34:50,167 INFO [RS:0;cn012:37560] hbase.ChoreService(323): Chore service for: cn012.l42scl.hortonworks.com,37560,1476973974890 had [[ScheduledChore: Name: MovedRegionsCleaner for region cn012.l42scl.hortonworks.com,37560,1476973974890 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS]] on shutdown 2016-10-20 14:34:50,168 INFO [RS:0;cn012:37560] regionserver.CompactSplitThread(403): Waiting for Split Thread to finish... 2016-10-20 14:34:50,168 INFO [RS:0;cn012:37560] regionserver.CompactSplitThread(403): Waiting for Merge Thread to finish... 2016-10-20 14:34:50,168 INFO [RS:0;cn012:37560] regionserver.CompactSplitThread(403): Waiting for Large Compaction Thread to finish... 2016-10-20 14:34:50,168 INFO [RS:0;cn012:37560] regionserver.CompactSplitThread(403): Waiting for Small Compaction Thread to finish... 2016-10-20 14:34:50,177 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/replication/rs/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:50,177 INFO [RS:0;cn012:37560] ipc.RpcServer(2336): Stopping server on 37560 2016-10-20 14:34:50,180 INFO [RpcServer.listener,port=37560] ipc.RpcServer$Listener(816): RpcServer.listener,port=37560: stopping 2016-10-20 14:34:50,180 INFO [RpcServer.responder] ipc.RpcServer$Responder(1059): RpcServer.responder: stopped 2016-10-20 14:34:50,180 INFO [RpcServer.responder] ipc.RpcServer$Responder(962): RpcServer.responder: stopping 2016-10-20 14:34:50,185 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rs/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:50,185 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rs/cn012.l42scl.hortonworks.com,37560,1476973974890 2016-10-20 14:34:50,185 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): regionserver:37560-0x157e2823ec60001, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rs 2016-10-20 14:34:50,186 INFO [main-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [cn012.l42scl.hortonworks.com,37560,1476973974890] 2016-10-20 14:34:50,189 INFO [main-EventThread] master.ServerManager(609): Cluster shutdown set; cn012.l42scl.hortonworks.com,37560,1476973974890 expired; onlineServers=1 2016-10-20 14:34:50,189 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/1/rs 2016-10-20 14:34:50,194 INFO [RS:0;cn012:37560] regionserver.HRegionServer(1135): stopping server cn012.l42scl.hortonworks.com,37560,1476973974890; zookeeper connection closed. 2016-10-20 14:34:50,194 INFO [M:0;cn012:41289] master.ServerManager(562): ZK shows there is only the master self online, exiting now 2016-10-20 14:34:50,194 DEBUG [M:0;cn012:41289] master.HMaster(1133): Stopping service threads 2016-10-20 14:34:50,194 INFO [RS:0;cn012:37560] regionserver.HRegionServer(1138): RS:0;cn012:37560 exiting 2016-10-20 14:34:50,194 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@47aef1f5] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(190): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@47aef1f5 2016-10-20 14:34:50,195 INFO [main] util.JVMClusterUtil(317): Shutdown of 1 master(s) and 1 regionserver(s) complete 2016-10-20 14:34:50,202 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/master 2016-10-20 14:34:50,202 INFO [M:0;cn012:41289] hbase.ChoreService(323): Chore service for: cn012.l42scl.hortonworks.com,41289,1476973974467_splitLogManager_ had [] on shutdown 2016-10-20 14:34:50,202 INFO [M:0;cn012:41289] flush.MasterFlushTableProcedureManager(78): stop: server shutting down. 2016-10-20 14:34:50,202 INFO [M:0;cn012:41289] master.LogRollMasterProcedureManager(55): stop: server shutting down. 2016-10-20 14:34:50,202 DEBUG [main-EventThread] zookeeper.ZKUtil(367): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Set watcher on znode that does not yet exist, /1/master 2016-10-20 14:34:50,203 INFO [M:0;cn012:41289] ipc.RpcServer(2336): Stopping server on 41289 2016-10-20 14:34:50,204 INFO [RpcServer.listener,port=41289] ipc.RpcServer$Listener(816): RpcServer.listener,port=41289: stopping 2016-10-20 14:34:50,204 INFO [RpcServer.responder] ipc.RpcServer$Responder(1059): RpcServer.responder: stopped 2016-10-20 14:34:50,205 INFO [RpcServer.responder] ipc.RpcServer$Responder(962): RpcServer.responder: stopping 2016-10-20 14:34:50,210 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): master:41289-0x157e2823ec60000, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/1/rs/cn012.l42scl.hortonworks.com,41289,1476973974467 2016-10-20 14:34:50,210 INFO [main-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [cn012.l42scl.hortonworks.com,41289,1476973974467] 2016-10-20 14:34:50,219 INFO [M:0;cn012:41289] regionserver.HRegionServer(1135): stopping server cn012.l42scl.hortonworks.com,41289,1476973974467; zookeeper connection closed. 2016-10-20 14:34:50,219 INFO [M:0;cn012:41289] regionserver.HRegionServer(1138): M:0;cn012:41289 exiting 2016-10-20 14:34:50,222 INFO [main] zookeeper.MiniZooKeeperCluster(319): Shutdown MiniZK cluster with all ZK servers 2016-10-20 14:34:50,222 WARN [main] datanode.DirectoryScanner(378): DirectoryScanner: shutdown has been called 2016-10-20 14:34:50,234 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2016-10-20 14:34:50,320 DEBUG [cn012:34094.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(590): replicationLogCleaner-0x157e2823ec6000b, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2016-10-20 14:34:50,320 DEBUG [cn012:34094.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x56e7f655-0x157e2823ec6000c, quorum=localhost:56750, baseZNode=/2 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2016-10-20 14:34:50,320 DEBUG [cn012:34094.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(679): hconnection-0x56e7f655-0x157e2823ec6000c, quorum=localhost:56750, baseZNode=/2 Received Disconnected from ZooKeeper, ignoring 2016-10-20 14:34:50,320 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x2dfb885e-0x157e2823ec6000f, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2016-10-20 14:34:50,320 DEBUG [cn012:41289.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(590): hconnection-0x3a112c11-0x157e2823ec60005, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2016-10-20 14:34:50,321 DEBUG [cn012:41289.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(679): hconnection-0x3a112c11-0x157e2823ec60005, quorum=localhost:56750, baseZNode=/1 Received Disconnected from ZooKeeper, ignoring 2016-10-20 14:34:50,320 DEBUG [cn012:34094.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(679): replicationLogCleaner-0x157e2823ec6000b, quorum=localhost:56750, baseZNode=/2 Received Disconnected from ZooKeeper, ignoring 2016-10-20 14:34:50,320 DEBUG [cn012:41289.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(590): replicationLogCleaner-0x157e2823ec60004, quorum=localhost:56750, baseZNode=/1 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2016-10-20 14:34:50,321 DEBUG [cn012:41289.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(679): replicationLogCleaner-0x157e2823ec60004, quorum=localhost:56750, baseZNode=/1 Received Disconnected from ZooKeeper, ignoring 2016-10-20 14:34:50,321 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(679): hconnection-0x2dfb885e-0x157e2823ec6000f, quorum=localhost:56750, baseZNode=/1 Received Disconnected from ZooKeeper, ignoring 2016-10-20 14:34:50,340 WARN [DataNode: [[[DISK]file:/mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/dfscluster_6b2a65f6-e6e0-418a-b6b5-3f3b53b4377b/dfs/data/data1/, [DISK]file:/mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/dfscluster_6b2a65f6-e6e0-418a-b6b5-3f3b53b4377b/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:37879] datanode.BPServiceActor(704): BPOfferService for Block pool BP-1003445509-172.18.128.12-1476973970079 (Datanode Uuid 1e9f9ed6-67b6-46b4-90c6-7f57e5b27722) service to localhost/127.0.0.1:37879 interrupted 2016-10-20 14:34:50,342 WARN [DataNode: [[[DISK]file:/mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/dfscluster_6b2a65f6-e6e0-418a-b6b5-3f3b53b4377b/dfs/data/data1/, [DISK]file:/mnt/disk2/a/7912/hbase-server/target/test-data/e9f85e68-3c25-4f9f-abe6-e9604b58e705/dfscluster_6b2a65f6-e6e0-418a-b6b5-3f3b53b4377b/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:37879] datanode.BPServiceActor(834): Ending block pool service for: Block pool BP-1003445509-172.18.128.12-1476973970079 (Datanode Uuid 1e9f9ed6-67b6-46b4-90c6-7f57e5b27722) service to localhost/127.0.0.1:37879 2016-10-20 14:34:50,362 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2016-10-20 14:34:50,501 INFO [main] hbase.HBaseTestingUtility(1156): Minicluster is down 2016-10-20 14:34:50,501 INFO [main] hbase.HBaseTestingUtility(2511): Stopping mini mapreduce cluster... 2016-10-20 14:34:50,529 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@cn012.l42scl.hortonworks.com:0 2016-10-20 14:34:50,740 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@cn012.l42scl.hortonworks.com:0 2016-10-20 14:34:50,884 ERROR [Thread[Thread-641,5,main]] delegation.AbstractDelegationTokenSecretManager$ExpiredTokenRemover(659): ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted 2016-10-20 14:34:50,889 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@cn012.l42scl.hortonworks.com:0 2016-10-20 14:34:51,009 WARN [ApplicationMaster Launcher] amlauncher.ApplicationMasterLauncher$LauncherThread(122): org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread interrupted. Returning. 2016-10-20 14:34:51,017 ERROR [ResourceManager Event Processor] resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor(675): Returning, interrupted : java.lang.InterruptedException 2016-10-20 14:34:51,020 ERROR [Thread[Thread-472,5,main]] delegation.AbstractDelegationTokenSecretManager$ExpiredTokenRemover(659): ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted 2016-10-20 14:34:51,035 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@cn012.l42scl.hortonworks.com:0 2016-10-20 14:34:51,140 ERROR [Thread[Thread-451,5,main]] delegation.AbstractDelegationTokenSecretManager$ExpiredTokenRemover(659): ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted 2016-10-20 14:34:51,141 INFO [main] hbase.HBaseTestingUtility(2514): Mini mapreduce cluster stopped 2016-10-20 14:34:51,153 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@1f97cf0d 2016-10-20 14:34:51,153 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2016-10-20 14:34:51,153 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@1f97cf0d 2016-10-20 14:34:51,154 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2016-10-20 14:34:51,155 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@1f97cf0d 2016-10-20 14:34:51,155 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2016-10-20 14:34:51,155 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@1f97cf0d 2016-10-20 14:34:51,155 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(120): Starting fs shutdown hook thread. 2016-10-20 14:34:51,171 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished.