2019-02-14 22:35:34,054 DEBUG [main] hbase.HBaseTestingUtility(348): Setting hbase.rootdir to /data/src/hbase/hbase-server/target/test-data/5a4d94bc-56f3-4901-bee1-e276bd3b8f91 2019-02-14 22:35:34,059 INFO [main] hbase.HBaseTestingUtility(1044): Starting up minicluster with 1 master(s) and 1 regionserver(s) and 1 datanode(s) 2019-02-14 22:35:34,059 INFO [main] hbase.HBaseTestingUtility(521): Created new mini-cluster data directory: /data/src/hbase/hbase-server/target/test-data/5a4d94bc-56f3-4901-bee1-e276bd3b8f91/dfscluster_af8fd26e-e004-4f4b-9256-660ce931a171, deleteOnExit=true 2019-02-14 22:35:34,059 INFO [main] hbase.HBaseTestingUtility(1059): STARTING DFS 2019-02-14 22:35:34,060 INFO [main] hbase.HBaseTestingUtility(759): Setting test.cache.data to /data/src/hbase/hbase-server/target/test-data/5a4d94bc-56f3-4901-bee1-e276bd3b8f91/cache_data in system properties and HBase conf 2019-02-14 22:35:34,061 INFO [main] hbase.HBaseTestingUtility(759): Setting hadoop.tmp.dir to /data/src/hbase/hbase-server/target/test-data/5a4d94bc-56f3-4901-bee1-e276bd3b8f91/hadoop_tmp in system properties and HBase conf 2019-02-14 22:35:34,061 INFO [main] hbase.HBaseTestingUtility(759): Setting hadoop.log.dir to /data/src/hbase/hbase-server/target/test-data/5a4d94bc-56f3-4901-bee1-e276bd3b8f91/hadoop_logs in system properties and HBase conf 2019-02-14 22:35:34,061 INFO [main] hbase.HBaseTestingUtility(759): Setting mapreduce.cluster.local.dir to /data/src/hbase/hbase-server/target/test-data/5a4d94bc-56f3-4901-bee1-e276bd3b8f91/mapred_local in system properties and HBase conf 2019-02-14 22:35:34,062 INFO [main] hbase.HBaseTestingUtility(759): Setting mapreduce.cluster.temp.dir to /data/src/hbase/hbase-server/target/test-data/5a4d94bc-56f3-4901-bee1-e276bd3b8f91/mapred_temp in system properties and HBase conf 2019-02-14 22:35:34,062 INFO [main] hbase.HBaseTestingUtility(750): read short circuit is OFF 2019-02-14 22:35:34,161 WARN [main] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2019-02-14 22:35:34,411 DEBUG [main] fs.HFileSystem(250): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2019-02-14 22:35:35,234 WARN [main] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2019-02-14 22:35:35,363 INFO [main] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2019-02-14 22:35:35,405 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2019-02-14 22:35:35,423 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/apurtell/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to /tmp/Jetty_localhost_38013_hdfs____.qfvf61/webapp 2019-02-14 22:35:35,533 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38013 2019-02-14 22:35:35,977 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2019-02-14 22:35:35,980 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/apurtell/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_43139_datanode____f0o38f/webapp 2019-02-14 22:35:36,062 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43139 2019-02-14 22:35:36,161 WARN [main] logging.Slf4JLogger(136): Failed to find a usable hardware address from the network interfaces; using random bytes: ab:e5:bd:d7:80:d3:ee:58 2019-02-14 22:35:36,733 ERROR [DataNode: [[[DISK]file:/data/src/hbase/hbase-server/target/test-data/5a4d94bc-56f3-4901-bee1-e276bd3b8f91/dfscluster_af8fd26e-e004-4f4b-9256-660ce931a171/dfs/data/data1/, [DISK]file:/data/src/hbase/hbase-server/target/test-data/5a4d94bc-56f3-4901-bee1-e276bd3b8f91/dfscluster_af8fd26e-e004-4f4b-9256-660ce931a171/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:41444] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2019-02-14 22:35:36,787 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x80aad690bceb15: from storage DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c node DatanodeRegistration(127.0.0.1:38317, datanodeUuid=b9a2f5fe-b4d7-4d90-97aa-c764a9863eae, infoPort=46683, infoSecurePort=0, ipcPort=36057, storageInfo=lv=-56;cid=testClusterID;nsid=301772869;c=0), blocks: 0, hasStaleStorage: true, processing time: 2 msecs 2019-02-14 22:35:36,787 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x80aad690bceb15: from storage DS-f3ca40dc-91f2-42ec-82fe-8f7a3a14736f node DatanodeRegistration(127.0.0.1:38317, datanodeUuid=b9a2f5fe-b4d7-4d90-97aa-c764a9863eae, infoPort=46683, infoSecurePort=0, ipcPort=36057, storageInfo=lv=-56;cid=testClusterID;nsid=301772869;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2019-02-14 22:35:36,794 DEBUG [main] hbase.HBaseTestingUtility(688): Setting hbase.rootdir to /data/src/hbase/hbase-server/target/test-data/5a4d94bc-56f3-4901-bee1-e276bd3b8f91 2019-02-14 22:35:36,835 ERROR [main] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2019-02-14 22:35:36,845 INFO [main] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran successful 'stat' on client port=50587 2019-02-14 22:35:36,861 INFO [main] fs.HFileSystem(281): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2019-02-14 22:35:36,864 INFO [main] fs.HFileSystem(281): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2019-02-14 22:35:37,134 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 7 2019-02-14 22:35:37,540 INFO [main] util.FSUtils(510): Created version file at hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862 with version=8 2019-02-14 22:35:37,699 INFO [main] client.ConnectionUtils(104): master/ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:0 server-side HConnection retries=350 2019-02-14 22:35:37,710 INFO [main] ipc.RpcExecutor(165): RpcExecutor name using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2019-02-14 22:35:37,711 INFO [main] ipc.RpcExecutor(165): RpcExecutor name using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2019-02-14 22:35:37,711 INFO [main] ipc.RpcExecutor(165): RpcExecutor name using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2019-02-14 22:35:37,715 INFO [main] ipc.RpcServer$Listener(682): master/ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:0: started 3 reader(s) listening on port=40925 2019-02-14 22:35:37,733 INFO [main] metrics.MetricRegistriesLoader(65): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2019-02-14 22:35:37,740 DEBUG [main] util.ClassSize(198): Using Unsafe to estimate memory layout 2019-02-14 22:35:37,751 INFO [main] fs.HFileSystem(281): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2019-02-14 22:35:37,754 INFO [main] fs.HFileSystem(281): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2019-02-14 22:35:37,804 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=master:40925 connecting to ZooKeeper ensemble=localhost:50587 2019-02-14 22:35:37,836 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:409250x0, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2019-02-14 22:35:37,837 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(678): master:40925-0x168ee26a5f90000 connected 2019-02-14 22:35:37,878 DEBUG [main] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2019-02-14 22:35:37,879 DEBUG [main] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2019-02-14 22:35:37,887 INFO [RpcServer.responder] ipc.RpcServer$Responder(1017): RpcServer.responder: starting 2019-02-14 22:35:37,887 INFO [RpcServer.listener,port=40925] ipc.RpcServer$Listener(828): RpcServer.listener,port=40925: starting 2019-02-14 22:35:37,887 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40925 2019-02-14 22:35:37,888 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40925 2019-02-14 22:35:37,888 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40925 2019-02-14 22:35:37,889 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=40925 2019-02-14 22:35:37,889 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925 2019-02-14 22:35:37,889 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.priority.FPBQ.Fifo.handler=0,queue=0,port=40925 2019-02-14 22:35:37,890 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=40925 2019-02-14 22:35:37,890 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.priority.FPBQ.Fifo.handler=2,queue=0,port=40925 2019-02-14 22:35:37,890 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=40925 2019-02-14 22:35:37,891 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=40925 2019-02-14 22:35:37,891 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=40925 2019-02-14 22:35:37,891 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=40925 2019-02-14 22:35:37,892 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=40925 2019-02-14 22:35:37,892 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=40925 2019-02-14 22:35:37,898 INFO [main] master.HMaster(454): hbase.rootdir=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862, hbase.cluster.distributed=false 2019-02-14 22:35:37,906 INFO [main] master.HMaster(2012): Adding backup master ZNode /hbase/backup-masters/ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583 2019-02-14 22:35:37,913 DEBUG [main] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583 2019-02-14 22:35:37,940 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2019-02-14 22:35:37,941 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2019-02-14 22:35:37,943 INFO [ip-172-31-47-35:40925.activeMasterManager] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583 from backup master directory 2019-02-14 22:35:37,945 DEBUG [main-EventThread] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2019-02-14 22:35:37,946 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2019-02-14 22:35:37,948 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583 2019-02-14 22:35:37,949 WARN [ip-172-31-47-35:40925.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2019-02-14 22:35:37,949 INFO [ip-172-31-47-35:40925.activeMasterManager] master.ActiveMasterManager(181): Registered Active Master=ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583 2019-02-14 22:35:37,985 INFO [main] client.ConnectionUtils(104): regionserver/ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:0 server-side HConnection retries=350 2019-02-14 22:35:37,985 INFO [main] ipc.RpcExecutor(165): RpcExecutor name using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2019-02-14 22:35:37,985 INFO [main] ipc.RpcExecutor(165): RpcExecutor name using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2019-02-14 22:35:37,986 INFO [main] ipc.RpcExecutor(165): RpcExecutor name using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2019-02-14 22:35:37,987 INFO [main] ipc.RpcServer$Listener(682): regionserver/ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:0: started 3 reader(s) listening on port=39948 2019-02-14 22:35:37,991 INFO [main] fs.HFileSystem(281): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2019-02-14 22:35:37,994 INFO [main] fs.HFileSystem(281): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2019-02-14 22:35:37,996 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:39948 connecting to ZooKeeper ensemble=localhost:50587 2019-02-14 22:35:38,003 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:399480x0, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2019-02-14 22:35:38,005 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(678): regionserver:39948-0x168ee26a5f90001 connected 2019-02-14 22:35:38,005 DEBUG [main] zookeeper.ZKUtil(374): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2019-02-14 22:35:38,006 DEBUG [main] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2019-02-14 22:35:38,007 INFO [RpcServer.responder] ipc.RpcServer$Responder(1017): RpcServer.responder: starting 2019-02-14 22:35:38,007 INFO [RpcServer.listener,port=39948] ipc.RpcServer$Listener(828): RpcServer.listener,port=39948: starting 2019-02-14 22:35:38,007 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39948 2019-02-14 22:35:38,009 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39948 2019-02-14 22:35:38,009 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39948 2019-02-14 22:35:38,010 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=39948 2019-02-14 22:35:38,010 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=39948 2019-02-14 22:35:38,011 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.priority.FPBQ.Fifo.handler=0,queue=0,port=39948 2019-02-14 22:35:38,011 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=39948 2019-02-14 22:35:38,012 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.priority.FPBQ.Fifo.handler=2,queue=0,port=39948 2019-02-14 22:35:38,012 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=39948 2019-02-14 22:35:38,013 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948 2019-02-14 22:35:38,013 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39948 2019-02-14 22:35:38,014 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=39948 2019-02-14 22:35:38,014 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=39948 2019-02-14 22:35:38,015 DEBUG [main] ipc.RpcExecutor(235): Started RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=39948 2019-02-14 22:35:38,065 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741826_1002{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f3ca40dc-91f2-42ec-82fe-8f7a3a14736f:NORMAL:127.0.0.1:38317|RBW]]} size 42 2019-02-14 22:35:38,081 INFO [M:0;ip-172-31-47-35:40925] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x5774ef9b connecting to ZooKeeper ensemble=localhost:50587 2019-02-14 22:35:38,081 INFO [RS:0;ip-172-31-47-35:39948] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x173a5a42 connecting to ZooKeeper ensemble=localhost:50587 2019-02-14 22:35:38,089 DEBUG [M:0;ip-172-31-47-35:40925-EventThread] zookeeper.ZooKeeperWatcher(615): hconnection-0x5774ef9b0x0, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2019-02-14 22:35:38,090 DEBUG [M:0;ip-172-31-47-35:40925-EventThread] zookeeper.ZooKeeperWatcher(678): hconnection-0x5774ef9b-0x168ee26a5f90002 connected 2019-02-14 22:35:38,091 DEBUG [RS:0;ip-172-31-47-35:39948-EventThread] zookeeper.ZooKeeperWatcher(615): hconnection-0x173a5a420x0, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2019-02-14 22:35:38,092 DEBUG [RS:0;ip-172-31-47-35:39948-EventThread] zookeeper.ZooKeeperWatcher(678): hconnection-0x173a5a42-0x168ee26a5f90003 connected 2019-02-14 22:35:38,092 INFO [M:0;ip-172-31-47-35:40925] client.ZooKeeperRegistry(107): ClusterId read in ZooKeeper is null 2019-02-14 22:35:38,092 INFO [RS:0;ip-172-31-47-35:39948] client.ZooKeeperRegistry(107): ClusterId read in ZooKeeper is null 2019-02-14 22:35:38,092 DEBUG [M:0;ip-172-31-47-35:40925] client.ConnectionManager$HConnectionImplementation(954): clusterid came back null, using default default-cluster 2019-02-14 22:35:38,093 DEBUG [RS:0;ip-172-31-47-35:39948] client.ConnectionManager$HConnectionImplementation(954): clusterid came back null, using default default-cluster 2019-02-14 22:35:38,110 DEBUG [M:0;ip-172-31-47-35:40925] ipc.AbstractRpcClient(199): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e8d7ba2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2019-02-14 22:35:38,110 DEBUG [RS:0;ip-172-31-47-35:39948] ipc.AbstractRpcClient(199): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58c0acee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2019-02-14 22:35:38,469 DEBUG [ip-172-31-47-35:40925.activeMasterManager] util.FSUtils(662): Created cluster ID file at hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/hbase.id with ID: ac8c378e-4811-4090-867f-bd3476d35b81 2019-02-14 22:35:38,503 INFO [ip-172-31-47-35:40925.activeMasterManager] master.MasterFileSystem(584): BOOTSTRAP: creating hbase:meta region 2019-02-14 22:35:38,504 INFO [ip-172-31-47-35:40925.activeMasterManager] regionserver.HRegion(6831): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', BLOOMFILTER => 'NONE', VERSIONS => '3', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'false', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862 Table name == hbase:meta 2019-02-14 22:35:38,532 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f3ca40dc-91f2-42ec-82fe-8f7a3a14736f:NORMAL:127.0.0.1:38317|FINALIZED]]} size 0 2019-02-14 22:35:38,541 DEBUG [ip-172-31-47-35:40925.activeMasterManager] regionserver.HRegion(826): Instantiated hbase:meta,,1.1588230740 2019-02-14 22:35:38,582 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/meta/1588230740/info 2019-02-14 22:35:38,591 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(547): Allocating LruBlockCache size=995.60 MB, blockSize=64 KB 2019-02-14 22:35:38,596 DEBUG [StoreOpener-1588230740-1] hfile.CacheConfig(562): Trying to use Internal l2 cache 2019-02-14 22:35:38,596 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(242): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2019-02-14 22:35:38,603 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(149): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2019-02-14 22:35:38,609 INFO [StoreOpener-1588230740-1] regionserver.HStore(370): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, encoding=NONE, compression=NONE 2019-02-14 22:35:38,617 DEBUG [ip-172-31-47-35:40925.activeMasterManager] regionserver.HRegion(4352): Found 0 recovered edits file(s) under hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/meta/1588230740 2019-02-14 22:35:38,619 DEBUG [ip-172-31-47-35:40925.activeMasterManager] regionserver.HRegion(4352): Found 0 recovered edits file(s) under hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/hbase/meta/1588230740 2019-02-14 22:35:38,631 DEBUG [ip-172-31-47-35:40925.activeMasterManager] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2019-02-14 22:35:38,641 DEBUG [ip-172-31-47-35:40925.activeMasterManager] wal.WALSplitter(751): Wrote region seqId=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/hbase/meta/1588230740/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2019-02-14 22:35:38,641 INFO [ip-172-31-47-35:40925.activeMasterManager] regionserver.HRegion(977): Onlined 1588230740; next sequenceid=2 2019-02-14 22:35:38,641 DEBUG [ip-172-31-47-35:40925.activeMasterManager] regionserver.HRegion(1526): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2019-02-14 22:35:38,642 DEBUG [ip-172-31-47-35:40925.activeMasterManager] regionserver.HRegion(1553): Updates disabled for region hbase:meta,,1.1588230740 2019-02-14 22:35:38,643 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(954): Closed info 2019-02-14 22:35:38,643 INFO [ip-172-31-47-35:40925.activeMasterManager] regionserver.HRegion(1660): Closed hbase:meta,,1.1588230740 2019-02-14 22:35:38,679 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741828_1004{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 397 2019-02-14 22:35:39,084 DEBUG [ip-172-31-47-35:40925.activeMasterManager] util.FSTableDescriptors(660): Wrote descriptor into: hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2019-02-14 22:35:39,095 INFO [ip-172-31-47-35:40925.activeMasterManager] fs.HFileSystem(281): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2019-02-14 22:35:39,101 INFO [ip-172-31-47-35:40925.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2019-02-14 22:35:39,101 DEBUG [ip-172-31-47-35:40925.activeMasterManager] util.FSTableDescriptors(208): Fetching table descriptors from the filesystem. 2019-02-14 22:35:39,114 INFO [ip-172-31-47-35:40925.activeMasterManager] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x73978144 connecting to ZooKeeper ensemble=localhost:50587 2019-02-14 22:35:39,118 DEBUG [ip-172-31-47-35:40925.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(615): hconnection-0x739781440x0, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2019-02-14 22:35:39,119 DEBUG [ip-172-31-47-35:40925.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(678): hconnection-0x73978144-0x168ee26a5f90004 connected 2019-02-14 22:35:39,119 DEBUG [ip-172-31-47-35:40925.activeMasterManager] ipc.AbstractRpcClient(199): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6259e096, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2019-02-14 22:35:39,143 INFO [ip-172-31-47-35:40925.activeMasterManager] balancer.StochasticLoadBalancer(168): loading config 2019-02-14 22:35:39,150 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2019-02-14 22:35:39,152 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2019-02-14 22:35:39,155 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2019-02-14 22:35:39,156 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2019-02-14 22:35:39,177 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2019-02-14 22:35:39,177 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2019-02-14 22:35:39,178 INFO [ip-172-31-47-35:40925.activeMasterManager] master.HMaster(693): Server active/primary master=ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583, sessionid=0x168ee26a5f90000, setting cluster-up flag (Was=false) 2019-02-14 22:35:39,181 INFO [RS:0;ip-172-31-47-35:39948] regionserver.HRegionServer(856): ClusterId : ac8c378e-4811-4090-867f-bd3476d35b81 2019-02-14 22:35:39,181 INFO [M:0;ip-172-31-47-35:40925] regionserver.HRegionServer(856): ClusterId : ac8c378e-4811-4090-867f-bd3476d35b81 2019-02-14 22:35:39,188 DEBUG [RS:0;ip-172-31-47-35:39948] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2019-02-14 22:35:39,196 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.RecoverableZooKeeper(584): Node /hbase/flush-table-proc already exists 2019-02-14 22:35:39,198 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.RecoverableZooKeeper(584): Node /hbase/flush-table-proc/acquired already exists 2019-02-14 22:35:39,201 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.RecoverableZooKeeper(584): Node /hbase/flush-table-proc/reached already exists 2019-02-14 22:35:39,203 INFO [ip-172-31-47-35:40925.activeMasterManager] procedure.ZKProcedureUtil(270): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2019-02-14 22:35:39,204 DEBUG [RS:0;ip-172-31-47-35:39948] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2019-02-14 22:35:39,204 DEBUG [RS:0;ip-172-31-47-35:39948] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2019-02-14 22:35:39,204 DEBUG [ip-172-31-47-35:40925.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(248): Starting the controller for procedure member:ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583 2019-02-14 22:35:39,209 DEBUG [RS:0;ip-172-31-47-35:39948] zookeeper.RecoverableZooKeeper(584): Node /hbase/online-snapshot/acquired already exists 2019-02-14 22:35:39,213 INFO [ip-172-31-47-35:40925.activeMasterManager] procedure.ZKProcedureUtil(270): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2019-02-14 22:35:39,214 DEBUG [RS:0;ip-172-31-47-35:39948] zookeeper.RecoverableZooKeeper(584): Node /hbase/online-snapshot/abort already exists 2019-02-14 22:35:39,214 DEBUG [RS:0;ip-172-31-47-35:39948] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2019-02-14 22:35:39,215 DEBUG [ip-172-31-47-35:40925.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(248): Starting the controller for procedure member:ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583 2019-02-14 22:35:39,217 INFO [RS:0;ip-172-31-47-35:39948] regionserver.MemStoreFlusher(130): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2019-02-14 22:35:39,217 WARN [ip-172-31-47-35:40925.activeMasterManager] snapshot.SnapshotManager(295): Couldn't delete working snapshot directory: hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/.hbase-snapshot/.tmp 2019-02-14 22:35:39,223 INFO [RS:0;ip-172-31-47-35:39948] throttle.PressureAwareCompactionThroughputController(132): Compaction throughput configurations, higher bound: 100.00 MB/sec, lower bound 50.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2019-02-14 22:35:39,224 INFO [RS:0;ip-172-31-47-35:39948] regionserver.HRegionServer$CompactionChecker(1641): CompactionChecker runs every 1sec 2019-02-14 22:35:39,228 DEBUG [RS:0;ip-172-31-47-35:39948] ipc.AbstractRpcClient(199): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4dd8f3a7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:0 2019-02-14 22:35:39,230 DEBUG [RS:0;ip-172-31-47-35:39948] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:RS:0;ip-172-31-47-35:39948 2019-02-14 22:35:39,235 INFO [RS:0;ip-172-31-47-35:39948] regionserver.RegionServerCoprocessorHost(71): System coprocessor loading is enabled 2019-02-14 22:35:39,235 INFO [RS:0;ip-172-31-47-35:39948] regionserver.RegionServerCoprocessorHost(72): Table coprocessor loading is enabled 2019-02-14 22:35:39,236 INFO [RS:0;ip-172-31-47-35:39948] regionserver.HRegionServer(2456): reportForDuty to master=ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583 with port=39948, startcode=1550183737985 2019-02-14 22:35:39,245 INFO [ip-172-31-47-35:40925.activeMasterManager] master.MasterCoprocessorHost(114): System coprocessor loading is enabled 2019-02-14 22:35:39,246 DEBUG [ip-172-31-47-35:40925.activeMasterManager] executor.ExecutorService(101): Starting executor service name=MASTER_OPEN_REGION-ip-172-31-47-35:40925, corePoolSize=5, maxPoolSize=5 2019-02-14 22:35:39,246 DEBUG [ip-172-31-47-35:40925.activeMasterManager] executor.ExecutorService(101): Starting executor service name=MASTER_CLOSE_REGION-ip-172-31-47-35:40925, corePoolSize=5, maxPoolSize=5 2019-02-14 22:35:39,246 DEBUG [ip-172-31-47-35:40925.activeMasterManager] executor.ExecutorService(101): Starting executor service name=MASTER_SERVER_OPERATIONS-ip-172-31-47-35:40925, corePoolSize=5, maxPoolSize=5 2019-02-14 22:35:39,246 DEBUG [ip-172-31-47-35:40925.activeMasterManager] executor.ExecutorService(101): Starting executor service name=MASTER_META_SERVER_OPERATIONS-ip-172-31-47-35:40925, corePoolSize=5, maxPoolSize=5 2019-02-14 22:35:39,247 DEBUG [ip-172-31-47-35:40925.activeMasterManager] executor.ExecutorService(101): Starting executor service name=M_LOG_REPLAY_OPS-ip-172-31-47-35:40925, corePoolSize=10, maxPoolSize=10 2019-02-14 22:35:39,247 DEBUG [ip-172-31-47-35:40925.activeMasterManager] executor.ExecutorService(101): Starting executor service name=MASTER_TABLE_OPERATIONS-ip-172-31-47-35:40925, corePoolSize=1, maxPoolSize=1 2019-02-14 22:35:39,261 INFO [ip-172-31-47-35:40925.activeMasterManager] procedure2.ProcedureExecutor(475): Starting procedure executor threads=17 2019-02-14 22:35:39,262 INFO [ip-172-31-47-35:40925.activeMasterManager] wal.WALProcedureStore(315): Starting WAL Procedure Store lease recovery 2019-02-14 22:35:39,267 WARN [ip-172-31-47-35:40925.activeMasterManager] util.CommonFSUtils$StreamCapabilities(849): Your Hadoop installation does not include the StreamCapabilities class from HDFS-11644, so we will skip checking if any FSDataOutputStreams actually support hflush/hsync. If you are running on top of HDFS this probably just means you have an older version and this can be ignored. If you are running on top of an alternate FileSystem implementation you should manually verify that hflush and hsync are implemented; otherwise you risk data loss and hard to diagnose errors when our assumptions are violated. 2019-02-14 22:35:39,267 DEBUG [ip-172-31-47-35:40925.activeMasterManager] util.CommonFSUtils$StreamCapabilities(856): The first request to check for StreamCapabilities came from this stacktrace. java.lang.ClassNotFoundException: org.apache.hadoop.fs.StreamCapabilities at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.util.CommonFSUtils$StreamCapabilities.(CommonFSUtils.java:845) at org.apache.hadoop.hbase.util.CommonFSUtils.hasCapability(CommonFSUtils.java:883) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:869) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:328) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:496) at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1328) at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1232) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:785) at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:209) at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:2042) at java.lang.Thread.run(Thread.java:748) 2019-02-14 22:35:39,271 DEBUG [ip-172-31-47-35:40925.activeMasterManager] wal.WALProcedureStore(896): Roll new state log: 1 2019-02-14 22:35:39,272 INFO [ip-172-31-47-35:40925.activeMasterManager] wal.WALProcedureStore(344): Lease acquired for flushLogId: 1 2019-02-14 22:35:39,272 DEBUG [ip-172-31-47-35:40925.activeMasterManager] wal.WALProcedureStore(361): No state logs to replay. 2019-02-14 22:35:39,272 DEBUG [ip-172-31-47-35:40925.activeMasterManager] procedure2.ProcedureExecutor$1(281): load procedures maxProcId=0 2019-02-14 22:35:39,283 INFO [ip-172-31-47-35:40925.activeMasterManager] cleaner.CleanerChore$DirScanPool(87): Cleaner pool size is 4 2019-02-14 22:35:39,283 DEBUG [RS:0;ip-172-31-47-35:39948] ipc.RpcConnection(126): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2019-02-14 22:35:39,284 DEBUG [ip-172-31-47-35:40925.activeMasterManager] cleaner.CleanerChore(249): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2019-02-14 22:35:39,285 INFO [ip-172-31-47-35:40925.activeMasterManager] zookeeper.RecoverableZooKeeper(120): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:50587 2019-02-14 22:35:39,290 DEBUG [ip-172-31-47-35:40925.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(615): replicationLogCleaner0x0, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2019-02-14 22:35:39,290 DEBUG [ip-172-31-47-35:40925.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(678): replicationLogCleaner-0x168ee26a5f90005 connected 2019-02-14 22:35:39,296 DEBUG [ip-172-31-47-35:40925.activeMasterManager] cleaner.CleanerChore(249): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2019-02-14 22:35:39,296 INFO [ip-172-31-47-35:40925.activeMasterManager] cleaner.LogCleaner(154): Creating OldWALs cleaners with size=2 2019-02-14 22:35:39,299 DEBUG [ip-172-31-47-35:40925.activeMasterManager] cleaner.CleanerChore(249): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2019-02-14 22:35:39,301 DEBUG [ip-172-31-47-35:40925.activeMasterManager] cleaner.CleanerChore(249): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2019-02-14 22:35:39,302 DEBUG [ip-172-31-47-35:40925.activeMasterManager] cleaner.CleanerChore(249): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2019-02-14 22:35:39,302 DEBUG [ip-172-31-47-35:40925.activeMasterManager] cleaner.HFileCleaner(230): Starting hfile cleaner for large files: ip-172-31-47-35:40925.activeMasterManager-HFileCleaner.large.0-1550183739302 2019-02-14 22:35:39,303 DEBUG [RS:0;ip-172-31-47-35:39948] ipc.BlockingRpcConnection(431): Connecting to ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:40925 2019-02-14 22:35:39,303 DEBUG [ip-172-31-47-35:40925.activeMasterManager] cleaner.HFileCleaner(245): Starting hfile cleaner for small files: ip-172-31-47-35:40925.activeMasterManager-HFileCleaner.small.0-1550183739303 2019-02-14 22:35:39,308 DEBUG [RpcServer.listener,port=40925] ipc.RpcServer$Listener(939): RpcServer.listener,port=40925: connection from 172.31.47.35:55682; # active connections: 1 2019-02-14 22:35:39,309 INFO [ip-172-31-47-35:40925.activeMasterManager] master.ServerManager(1135): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=1 server(s), max=1 server(s), timeout=4500ms, lastChange=0ms 2019-02-14 22:35:39,310 INFO [RpcServer.reader=1,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=40925] ipc.RpcServer$Connection(1838): Auth successful for apurtell.hfs.0 (auth:SIMPLE) 2019-02-14 22:35:39,314 INFO [RpcServer.reader=1,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=40925] ipc.RpcServer$Connection(1868): Connection from 172.31.47.35 port: 55682 with version info: version: "1.5.0" url: "git://ip-172-31-47-35.us-west-2.compute.internal/data/src/hbase" revision: "" user: "apurtell" date: "Thu Feb 14 22:35:08 UTC 2019" src_checksum: "60dfaeb22217478e0331d7332d9bc018" version_major: 1 version_minor: 5 2019-02-14 22:35:39,325 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.ServerManager(477): Registering server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:39,334 INFO [RS:0;ip-172-31-47-35:39948] regionserver.HRegionServer(1463): Config from master: hbase.rootdir=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862 2019-02-14 22:35:39,334 INFO [RS:0;ip-172-31-47-35:39948] regionserver.HRegionServer(1463): Config from master: fs.defaultFS=hdfs://localhost:41444 2019-02-14 22:35:39,334 INFO [RS:0;ip-172-31-47-35:39948] regionserver.HRegionServer(1463): Config from master: hbase.master.info.port=-1 2019-02-14 22:35:39,338 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2019-02-14 22:35:39,338 DEBUG [RS:0;ip-172-31-47-35:39948] zookeeper.ZKUtil(374): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:39,339 WARN [RS:0;ip-172-31-47-35:39948] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2019-02-14 22:35:39,339 DEBUG [main-EventThread] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:39,339 INFO [RS:0;ip-172-31-47-35:39948] hfile.CacheConfig(267): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2019-02-14 22:35:39,340 DEBUG [RS:0;ip-172-31-47-35:39948] regionserver.HRegionServer(1745): logDir=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/WALs/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:39,340 DEBUG [main-EventThread] zookeeper.RegionServerTracker(96): Added tracking of RS /hbase/rs/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:39,352 DEBUG [RS:0;ip-172-31-47-35:39948] regionserver.Replication(160): ReplicationStatisticsThread 300 2019-02-14 22:35:39,358 INFO [RS:0;ip-172-31-47-35:39948] wal.WALFactory(144): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2019-02-14 22:35:39,360 INFO [ip-172-31-47-35:40925.activeMasterManager] master.ServerManager(1152): Finished wait on RegionServer count=1; waited=50ms, expected min=1 server(s), max=1 server(s), master is running 2019-02-14 22:35:39,361 INFO [RS:0;ip-172-31-47-35:39948] regionserver.MetricsRegionServerWrapperImpl(109): Computing regionserver metrics every 5000 milliseconds 2019-02-14 22:35:39,362 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:39,368 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:39,371 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:39,379 DEBUG [RS:0;ip-172-31-47-35:39948] executor.ExecutorService(101): Starting executor service name=RS_OPEN_REGION-ip-172-31-47-35:39948, corePoolSize=3, maxPoolSize=3 2019-02-14 22:35:39,380 DEBUG [RS:0;ip-172-31-47-35:39948] executor.ExecutorService(101): Starting executor service name=RS_OPEN_META-ip-172-31-47-35:39948, corePoolSize=1, maxPoolSize=1 2019-02-14 22:35:39,380 DEBUG [RS:0;ip-172-31-47-35:39948] executor.ExecutorService(101): Starting executor service name=RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948, corePoolSize=3, maxPoolSize=3 2019-02-14 22:35:39,380 DEBUG [RS:0;ip-172-31-47-35:39948] executor.ExecutorService(101): Starting executor service name=RS_CLOSE_REGION-ip-172-31-47-35:39948, corePoolSize=3, maxPoolSize=3 2019-02-14 22:35:39,380 DEBUG [RS:0;ip-172-31-47-35:39948] executor.ExecutorService(101): Starting executor service name=RS_CLOSE_META-ip-172-31-47-35:39948, corePoolSize=1, maxPoolSize=1 2019-02-14 22:35:39,380 DEBUG [RS:0;ip-172-31-47-35:39948] executor.ExecutorService(101): Starting executor service name=RS_LOG_REPLAY_OPS-ip-172-31-47-35:39948, corePoolSize=2, maxPoolSize=2 2019-02-14 22:35:39,380 DEBUG [RS:0;ip-172-31-47-35:39948] executor.ExecutorService(101): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-ip-172-31-47-35:39948, corePoolSize=10, maxPoolSize=10 2019-02-14 22:35:39,380 DEBUG [RS:0;ip-172-31-47-35:39948] executor.ExecutorService(101): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-ip-172-31-47-35:39948, corePoolSize=3, maxPoolSize=3 2019-02-14 22:35:39,383 DEBUG [RS:0;ip-172-31-47-35:39948] zookeeper.ZKUtil(374): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:39,383 INFO [RS:0;ip-172-31-47-35:39948] regionserver.ReplicationSourceManager(253): Current list of replicators: [ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985] other RSs: [ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985] 2019-02-14 22:35:39,433 INFO [SplitLogWorker-ip-172-31-47-35:39948] regionserver.SplitLogWorker(140): SplitLogWorker ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 starting 2019-02-14 22:35:39,434 INFO [RS:0;ip-172-31-47-35:39948] regionserver.HeapMemoryManager(205): Starting HeapMemoryTuner chore. 2019-02-14 22:35:39,436 INFO [RS:0;ip-172-31-47-35:39948] regionserver.HRegionServer(1497): Serving as ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985, RpcServer on ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:39948, sessionid=0x168ee26a5f90001 2019-02-14 22:35:39,437 DEBUG [RS:0;ip-172-31-47-35:39948] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2019-02-14 22:35:39,437 DEBUG [RS:0;ip-172-31-47-35:39948] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:39,437 DEBUG [RS:0;ip-172-31-47-35:39948] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' 2019-02-14 22:35:39,437 DEBUG [RS:0;ip-172-31-47-35:39948] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:39,437 DEBUG [RS:0;ip-172-31-47-35:39948] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:39,438 DEBUG [RS:0;ip-172-31-47-35:39948] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2019-02-14 22:35:39,438 DEBUG [RS:0;ip-172-31-47-35:39948] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2019-02-14 22:35:39,438 DEBUG [RS:0;ip-172-31-47-35:39948] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:39,438 DEBUG [RS:0;ip-172-31-47-35:39948] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' 2019-02-14 22:35:39,438 DEBUG [RS:0;ip-172-31-47-35:39948] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2019-02-14 22:35:39,438 DEBUG [RS:0;ip-172-31-47-35:39948] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2019-02-14 22:35:39,439 DEBUG [RS:0;ip-172-31-47-35:39948] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2019-02-14 22:35:39,443 INFO [RS:0;ip-172-31-47-35:39948] quotas.RegionServerQuotaManager(55): Quota support disabled 2019-02-14 22:35:39,572 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:39,773 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:39,973 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,174 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,174 INFO [ip-172-31-47-35:40925.activeMasterManager] zookeeper.MetaTableLocator(533): Deleting hbase:meta region location in ZooKeeper 2019-02-14 22:35:40,179 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.RecoverableZooKeeper(188): Node /hbase/meta-region-server already deleted, retry=false 2019-02-14 22:35:40,183 DEBUG [ip-172-31-47-35:40925.activeMasterManager] master.AssignmentManager(2608): No previous transition plan found (or ignoring an existing plan) for hbase:meta,,1.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985; 1 (online=1) available servers, forceNewPlan=false 2019-02-14 22:35:40,183 INFO [ip-172-31-47-35:40925.activeMasterManager] master.AssignmentManager(2211): Setting node as OFFLINED in ZooKeeper for region {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2019-02-14 22:35:40,183 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKAssign(205): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Creating (or updating) unassigned node 1588230740 with OFFLINE state 2019-02-14 22:35:40,189 DEBUG [ip-172-31-47-35:40925.activeMasterManager] master.AssignmentManager(2228): Setting table hbase:meta to ENABLED state. 2019-02-14 22:35:40,189 INFO [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKTableStateManager(100): Moving table hbase:meta state from null to ENABLED 2019-02-14 22:35:40,195 INFO [ip-172-31-47-35:40925.activeMasterManager] master.AssignmentManager(2243): Assigning hbase:meta,,1.1588230740 to ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:40,196 INFO [ip-172-31-47-35:40925.activeMasterManager] master.RegionStates(1257): Transition {1588230740 state=OFFLINE, ts=1550183740183, server=null} to {1588230740 state=PENDING_OPEN, ts=1550183740196, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:40,196 DEBUG [ip-172-31-47-35:40925.activeMasterManager] master.ServerManager(1046): New admin connection to ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:40,209 DEBUG [ip-172-31-47-35:40925.activeMasterManager] ipc.RpcConnection(126): Use SIMPLE authentication for service AdminService, sasl=false 2019-02-14 22:35:40,210 DEBUG [ip-172-31-47-35:40925.activeMasterManager] ipc.BlockingRpcConnection(431): Connecting to ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:39948 2019-02-14 22:35:40,212 DEBUG [RpcServer.listener,port=39948] ipc.RpcServer$Listener(939): RpcServer.listener,port=39948: connection from 172.31.47.35:51710; # active connections: 1 2019-02-14 22:35:40,213 INFO [RpcServer.reader=1,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Connection(1838): Auth successful for apurtell (auth:SIMPLE) 2019-02-14 22:35:40,213 INFO [RpcServer.reader=1,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Connection(1868): Connection from 172.31.47.35 port: 51710 with version info: version: "1.5.0" url: "git://ip-172-31-47-35.us-west-2.compute.internal/data/src/hbase" revision: "" user: "apurtell" date: "Thu Feb 14 22:35:08 UTC 2019" src_checksum: "60dfaeb22217478e0331d7332d9bc018" version_major: 1 version_minor: 5 2019-02-14 22:35:40,217 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=39948] regionserver.RSRpcServices(1840): Open hbase:meta,,1.1588230740 2019-02-14 22:35:40,220 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] zookeeper.ZKAssign(805): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioning 1588230740 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2019-02-14 22:35:40,223 DEBUG [ip-172-31-47-35:40925.activeMasterManager] procedure.MasterProcedureScheduler(387): Wake event ProcedureEvent(server crash processing) 2019-02-14 22:35:40,223 INFO [ip-172-31-47-35:40925.activeMasterManager] master.ServerManager(711): AssignmentManager hasn't finished failover cleanup; waiting 2019-02-14 22:35:40,224 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2019-02-14 22:35:40,224 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] zookeeper.ZKAssign(880): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioned node 1588230740 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2019-02-14 22:35:40,224 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,225 INFO [RS_OPEN_META-ip-172-31-47-35:39948-0] wal.WALFactory(144): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2019-02-14 22:35:40,225 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,226 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,226 DEBUG [AM.ZK.Worker-pool5-t1] master.AssignmentManager(1052): Handling RS_ZK_REGION_OPENING, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985, region=1588230740, current_state={1588230740 state=PENDING_OPEN, ts=1550183740196, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:40,226 INFO [AM.ZK.Worker-pool5-t1] master.RegionStates(1257): Transition {1588230740 state=PENDING_OPEN, ts=1550183740196, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} to {1588230740 state=OPENING, ts=1550183740226, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:40,228 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,228 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,229 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,231 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,231 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,232 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,234 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,235 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,236 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,236 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,237 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,238 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,238 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,239 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,239 INFO [RS_OPEN_META-ip-172-31-47-35:39948-0] wal.FSHLog(533): WAL configuration: blocksize=256 MB, rollsize=243.20 MB, prefix=ip-172-31-47-35.us-west-2.compute.internal%2C39948%2C1550183737985.meta, suffix=.meta, logDir=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/WALs/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985, archiveDir=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/oldWALs 2019-02-14 22:35:40,240 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,241 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,241 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,242 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,243 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,245 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,246 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,247 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,247 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,248 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,248 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,249 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,250 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,250 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,251 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,252 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,253 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,254 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,255 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,256 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,257 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,258 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,259 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,261 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,264 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,265 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,265 INFO [RS_OPEN_META-ip-172-31-47-35:39948-0] wal.FSHLog(1458): Slow sync cost: 12 ms, current pipeline: [] 2019-02-14 22:35:40,266 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,266 INFO [RS_OPEN_META-ip-172-31-47-35:39948-0] wal.FSHLog(911): New WAL /user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/WALs/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985/ip-172-31-47-35.us-west-2.compute.internal%2C39948%2C1550183737985.meta.1550183740240.meta 2019-02-14 22:35:40,266 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,267 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,268 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,268 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,269 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,270 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,270 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,271 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,272 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,272 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,273 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,274 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,274 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,275 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,276 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,276 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,277 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,278 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,278 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,279 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,279 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,280 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,281 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,281 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,282 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,283 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,283 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,284 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,284 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,285 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,285 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] regionserver.HRegion(7020): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2019-02-14 22:35:40,286 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,286 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,287 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,288 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,288 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,289 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,289 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,290 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,290 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,291 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,292 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,292 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,293 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,293 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,294 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,295 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,295 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,296 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,296 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,297 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,298 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,298 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,299 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,299 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,300 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,301 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,301 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,301 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] coprocessor.CoprocessorHost(206): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2019-02-14 22:35:40,302 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,303 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,303 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,304 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,305 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,305 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,306 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,306 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,307 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,308 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,308 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,309 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] regionserver.HRegion(8429): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2019-02-14 22:35:40,309 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,310 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,312 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,312 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,313 INFO [RS_OPEN_META-ip-172-31-47-35:39948-0] regionserver.RegionCoprocessorHost(379): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2019-02-14 22:35:40,313 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,314 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,314 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,315 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,315 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,316 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,317 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,318 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,319 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,319 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2019-02-14 22:35:40,319 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] regionserver.HRegion(826): Instantiated hbase:meta,,1.1588230740 2019-02-14 22:35:40,319 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,320 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,320 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,321 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,321 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,322 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,322 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,323 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,323 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,324 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,326 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,326 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,326 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/meta/1588230740/info 2019-02-14 22:35:40,326 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/meta/1588230740/info 2019-02-14 22:35:40,327 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,327 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(242): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2019-02-14 22:35:40,328 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(149): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2019-02-14 22:35:40,328 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,329 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,329 INFO [StoreOpener-1588230740-1] regionserver.HStore(370): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, encoding=NONE, compression=NONE 2019-02-14 22:35:40,330 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,332 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,332 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] regionserver.HRegion(4352): Found 0 recovered edits file(s) under hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/meta/1588230740 2019-02-14 22:35:40,333 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,333 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,334 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,334 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,335 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] regionserver.HRegion(4352): Found 0 recovered edits file(s) under hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/hbase/meta/1588230740 2019-02-14 22:35:40,335 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,336 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,337 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,337 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2019-02-14 22:35:40,337 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,338 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,338 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,339 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,340 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,340 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,340 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,341 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,342 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] wal.WALSplitter(751): Wrote region seqId=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/hbase/meta/1588230740/recovered.edits/3.seqid to file, newSeqId=3, maxSeqId=2 2019-02-14 22:35:40,342 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,342 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,343 INFO [RS_OPEN_META-ip-172-31-47-35:39948-0] regionserver.HRegion(977): Onlined 1588230740; next sequenceid=3 2019-02-14 22:35:40,343 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,343 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,344 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,344 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,345 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,345 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,345 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,346 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,346 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,347 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,347 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,348 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,348 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,348 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,349 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,349 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,350 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,350 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,351 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,351 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,352 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,352 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,353 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,353 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,354 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,354 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,355 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,355 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,356 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,356 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,358 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,358 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,359 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,359 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,360 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,360 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,361 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,362 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,362 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] zookeeper.ZKAssign(643): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Attempting to retransition opening state of node 1588230740 2019-02-14 22:35:40,362 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,363 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,363 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2058): Post open deploy tasks for hbase:meta,,1.1588230740 2019-02-14 22:35:40,363 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,364 INFO [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(447): Setting hbase:meta region location in ZooKeeper as ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:40,364 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,365 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,365 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,366 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,367 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,367 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,368 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,370 DEBUG [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(459): META region location doesn't exist, create it 2019-02-14 22:35:40,370 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2019-02-14 22:35:40,372 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2092): Finished post open deploy task for hbase:meta,,1.1588230740 2019-02-14 22:35:40,373 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] zookeeper.ZKAssign(805): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioning 1588230740 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2019-02-14 22:35:40,375 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] zookeeper.ZKAssign(880): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioned node 1588230740 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2019-02-14 22:35:40,375 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2019-02-14 22:35:40,375 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] coordination.ZkOpenRegionCoordination(91): Transitioned 1588230740 to OPENED in zk on ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:40,375 DEBUG [RS_OPEN_META-ip-172-31-47-35:39948-0] handler.OpenRegionHandler(172): Opened hbase:meta,,1.1588230740 on ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:40,376 DEBUG [AM.ZK.Worker-pool5-t2] master.AssignmentManager(1052): Handling RS_ZK_REGION_OPENED, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985, region=1588230740, current_state={1588230740 state=OPENING, ts=1550183740226, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:40,376 INFO [AM.ZK.Worker-pool5-t2] master.RegionStates(1257): Transition {1588230740 state=OPENING, ts=1550183740226, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} to {1588230740 state=OPEN, ts=1550183740376, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:40,377 INFO [AM.ZK.Worker-pool5-t2] coordination.ZkOpenRegionCoordination(348): Handling OPENED of 1588230740 from ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583; deleting unassigned node 2019-02-14 22:35:40,379 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2019-02-14 22:35:40,379 DEBUG [AM.ZK.Worker-pool5-t2] zookeeper.ZKAssign(479): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Deleted unassigned node 1588230740 in expected state RS_ZK_REGION_OPENED 2019-02-14 22:35:40,380 DEBUG [AM.ZK.Worker-pool5-t3] master.AssignmentManager$5(1448): Znode hbase:meta,,1.1588230740 deleted, state: {1588230740 state=OPEN, ts=1550183740376, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:40,381 DEBUG [AM.ZK.Worker-pool5-t3] master.RegionStates(553): Onlined 1588230740 on ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:40,382 INFO [ip-172-31-47-35:40925.activeMasterManager] master.HMaster(1079): hbase:meta with replicaId 0 assigned=1, rit=false, location=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:40,525 INFO [RS:0;ip-172-31-47-35:39948] wal.FSHLog(533): WAL configuration: blocksize=256 MB, rollsize=243.20 MB, prefix=ip-172-31-47-35.us-west-2.compute.internal%2C39948%2C1550183737985, suffix=, logDir=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/WALs/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985, archiveDir=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/oldWALs 2019-02-14 22:35:40,541 INFO [RS:0;ip-172-31-47-35:39948] wal.FSHLog(1458): Slow sync cost: 12 ms, current pipeline: [] 2019-02-14 22:35:40,542 INFO [RS:0;ip-172-31-47-35:39948] wal.FSHLog(911): New WAL /user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/WALs/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985/ip-172-31-47-35.us-west-2.compute.internal%2C39948%2C1550183737985.1550183740526 2019-02-14 22:35:40,558 DEBUG [hconnection-0x5774ef9b-shared--pool8-t1] ipc.RpcConnection(126): Use SIMPLE authentication for service ClientService, sasl=false 2019-02-14 22:35:40,558 DEBUG [hconnection-0x5774ef9b-shared--pool8-t1] ipc.BlockingRpcConnection(431): Connecting to ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:39948 2019-02-14 22:35:40,559 DEBUG [RpcServer.listener,port=39948] ipc.RpcServer$Listener(939): RpcServer.listener,port=39948: connection from 172.31.47.35:51716; # active connections: 2 2019-02-14 22:35:40,560 INFO [RpcServer.reader=2,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Connection(1838): Auth successful for apurtell (auth:SIMPLE) 2019-02-14 22:35:40,560 INFO [RpcServer.reader=2,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Connection(1868): Connection from 172.31.47.35 port: 51716 with version info: version: "1.5.0" url: "git://ip-172-31-47-35.us-west-2.compute.internal/data/src/hbase" revision: "" user: "apurtell" date: "Thu Feb 14 22:35:08 UTC 2019" src_checksum: "60dfaeb22217478e0331d7332d9bc018" version_major: 1 version_minor: 5 2019-02-14 22:35:40,579 INFO [ip-172-31-47-35:40925.activeMasterManager] hbase.MetaMigrationConvertingToPB(165): hbase:meta doesn't have any entries to update. 2019-02-14 22:35:40,579 INFO [ip-172-31-47-35:40925.activeMasterManager] hbase.MetaMigrationConvertingToPB(131): META already up-to date with PB serialization 2019-02-14 22:35:40,583 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKAssign(497): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Deleting any existing unassigned nodes 2019-02-14 22:35:40,584 INFO [ip-172-31-47-35:40925.activeMasterManager] master.AssignmentManager(691): Clean cluster startup. Assigning user regions 2019-02-14 22:35:40,585 INFO [ip-172-31-47-35:40925.activeMasterManager] master.AssignmentManager(547): Joined the cluster in 6ms, failover=false 2019-02-14 22:35:40,590 INFO [ip-172-31-47-35:40925.activeMasterManager] master.TableNamespaceManager(85): Namespace table not found. Creating... 2019-02-14 22:35:40,593 INFO [ip-172-31-47-35:40925.activeMasterManager] master.HMaster(1782): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2019-02-14 22:35:40,723 DEBUG [ip-172-31-47-35:40925.activeMasterManager] procedure2.ProcedureExecutor(769): Procedure CreateTableProcedure (table=hbase:namespace) id=1 owner=apurtell.hfs.0 state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2019-02-14 22:35:40,737 DEBUG [ProcedureExecutor-0] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/hbase:namespace/write-master:409250000000000 2019-02-14 22:35:40,863 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741832_1008{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f3ca40dc-91f2-42ec-82fe-8f7a3a14736f:NORMAL:127.0.0.1:38317|RBW]]} size 312 2019-02-14 22:35:41,266 DEBUG [ProcedureExecutor-0] util.FSTableDescriptors(660): Wrote descriptor into: hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2019-02-14 22:35:41,270 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6831): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/.tmp Table name == hbase:namespace 2019-02-14 22:35:41,285 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f3ca40dc-91f2-42ec-82fe-8f7a3a14736f:NORMAL:127.0.0.1:38317|RBW]]} size 0 2019-02-14 22:35:41,286 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(826): Instantiated hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5. 2019-02-14 22:35:41,287 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1526): Closing hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5.: disabling compactions & flushes 2019-02-14 22:35:41,287 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1553): Updates disabled for region hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5. 2019-02-14 22:35:41,287 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1660): Closed hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5. 2019-02-14 22:35:41,448 INFO [ProcedureExecutor-0] hbase.MetaTableAccessor(1234): Added 1 2019-02-14 22:35:41,553 INFO [ProcedureExecutor-0] zookeeper.ZKTableStateManager(100): Moving table hbase:namespace state from null to ENABLING 2019-02-14 22:35:41,558 INFO [ProcedureExecutor-0] master.AssignmentManager(1724): Assigning 1 region(s) to ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:41,560 DEBUG [ProcedureExecutor-0] zookeeper.ZKAssign(174): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Async create of unassigned node c71b20fd2578c13df17ccd07b9ab4dd5 with OFFLINE state 2019-02-14 22:35:41,562 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2019-02-14 22:35:41,563 DEBUG [main-EventThread] master.OfflineCallback(69): rs={c71b20fd2578c13df17ccd07b9ab4dd5 state=OFFLINE, ts=1550183741558, server=null}, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:41,563 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={c71b20fd2578c13df17ccd07b9ab4dd5 state=OFFLINE, ts=1550183741558, server=null}, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:41,565 DEBUG [ProcedureExecutor-0] master.AssignmentManager(1777): ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 unassigned znodes=1 of total=1; oldCounter=0 2019-02-14 22:35:41,565 INFO [ProcedureExecutor-0] master.RegionStates(1257): Transition {c71b20fd2578c13df17ccd07b9ab4dd5 state=OFFLINE, ts=1550183741560, server=null} to {c71b20fd2578c13df17ccd07b9ab4dd5 state=PENDING_OPEN, ts=1550183741565, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:41,567 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.RSRpcServices(1840): Open hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5. 2019-02-14 22:35:41,576 DEBUG [ProcedureExecutor-0] master.AssignmentManager(1919): Bulk assigning done for ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:41,576 DEBUG [RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948-0] zookeeper.ZKAssign(805): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioning c71b20fd2578c13df17ccd07b9ab4dd5 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2019-02-14 22:35:41,576 INFO [ProcedureExecutor-0] zookeeper.ZKTableStateManager(100): Moving table hbase:namespace state from ENABLING to ENABLED 2019-02-14 22:35:41,579 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/c71b20fd2578c13df17ccd07b9ab4dd5 2019-02-14 22:35:41,579 DEBUG [RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948-0] zookeeper.ZKAssign(880): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioned node c71b20fd2578c13df17ccd07b9ab4dd5 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2019-02-14 22:35:41,579 DEBUG [RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948-0] regionserver.HRegion(7020): Opening region: {ENCODED => c71b20fd2578c13df17ccd07b9ab4dd5, NAME => 'hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5.', STARTKEY => '', ENDKEY => ''} 2019-02-14 22:35:41,580 DEBUG [AM.ZK.Worker-pool5-t5] master.AssignmentManager(1052): Handling RS_ZK_REGION_OPENING, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985, region=c71b20fd2578c13df17ccd07b9ab4dd5, current_state={c71b20fd2578c13df17ccd07b9ab4dd5 state=PENDING_OPEN, ts=1550183741565, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:41,580 INFO [AM.ZK.Worker-pool5-t5] master.RegionStates(1257): Transition {c71b20fd2578c13df17ccd07b9ab4dd5 state=PENDING_OPEN, ts=1550183741565, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} to {c71b20fd2578c13df17ccd07b9ab4dd5 state=OPENING, ts=1550183741580, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:41,580 DEBUG [RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace c71b20fd2578c13df17ccd07b9ab4dd5 2019-02-14 22:35:41,581 DEBUG [RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948-0] regionserver.HRegion(826): Instantiated hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5. 2019-02-14 22:35:41,586 DEBUG [StoreOpener-c71b20fd2578c13df17ccd07b9ab4dd5-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/namespace/c71b20fd2578c13df17ccd07b9ab4dd5/info 2019-02-14 22:35:41,586 DEBUG [StoreOpener-c71b20fd2578c13df17ccd07b9ab4dd5-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/namespace/c71b20fd2578c13df17ccd07b9ab4dd5/info 2019-02-14 22:35:41,586 INFO [StoreOpener-c71b20fd2578c13df17ccd07b9ab4dd5-1] hfile.CacheConfig(242): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2019-02-14 22:35:41,586 INFO [StoreOpener-c71b20fd2578c13df17ccd07b9ab4dd5-1] compactions.CompactionConfiguration(149): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2019-02-14 22:35:41,588 INFO [StoreOpener-c71b20fd2578c13df17ccd07b9ab4dd5-1] regionserver.HStore(370): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, encoding=NONE, compression=NONE 2019-02-14 22:35:41,589 DEBUG [RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948-0] regionserver.HRegion(4352): Found 0 recovered edits file(s) under hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/namespace/c71b20fd2578c13df17ccd07b9ab4dd5 2019-02-14 22:35:41,590 DEBUG [RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948-0] regionserver.HRegion(4352): Found 0 recovered edits file(s) under hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/hbase/namespace/c71b20fd2578c13df17ccd07b9ab4dd5 2019-02-14 22:35:41,592 DEBUG [RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2019-02-14 22:35:41,597 DEBUG [RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948-0] wal.WALSplitter(751): Wrote region seqId=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/hbase/namespace/c71b20fd2578c13df17ccd07b9ab4dd5/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2019-02-14 22:35:41,597 INFO [RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948-0] regionserver.HRegion(977): Onlined c71b20fd2578c13df17ccd07b9ab4dd5; next sequenceid=2 2019-02-14 22:35:41,600 DEBUG [RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948-0] zookeeper.ZKAssign(643): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Attempting to retransition opening state of node c71b20fd2578c13df17ccd07b9ab4dd5 2019-02-14 22:35:41,601 INFO [PostOpenDeployTasks:c71b20fd2578c13df17ccd07b9ab4dd5] regionserver.HRegionServer(2058): Post open deploy tasks for hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5. 2019-02-14 22:35:41,607 INFO [PostOpenDeployTasks:c71b20fd2578c13df17ccd07b9ab4dd5] hbase.MetaTableAccessor(1434): Updated row hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5. with server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:41,608 DEBUG [PostOpenDeployTasks:c71b20fd2578c13df17ccd07b9ab4dd5] regionserver.HRegionServer(2092): Finished post open deploy task for hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5. 2019-02-14 22:35:41,608 DEBUG [RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948-0] zookeeper.ZKAssign(805): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioning c71b20fd2578c13df17ccd07b9ab4dd5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2019-02-14 22:35:41,611 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/c71b20fd2578c13df17ccd07b9ab4dd5 2019-02-14 22:35:41,611 DEBUG [RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948-0] zookeeper.ZKAssign(880): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioned node c71b20fd2578c13df17ccd07b9ab4dd5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2019-02-14 22:35:41,611 DEBUG [RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948-0] coordination.ZkOpenRegionCoordination(91): Transitioned c71b20fd2578c13df17ccd07b9ab4dd5 to OPENED in zk on ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:41,611 DEBUG [RS_OPEN_PRIORITY_REGION-ip-172-31-47-35:39948-0] handler.OpenRegionHandler(172): Opened hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5. on ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:41,612 DEBUG [AM.ZK.Worker-pool5-t6] master.AssignmentManager(1052): Handling RS_ZK_REGION_OPENED, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985, region=c71b20fd2578c13df17ccd07b9ab4dd5, current_state={c71b20fd2578c13df17ccd07b9ab4dd5 state=OPENING, ts=1550183741580, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:41,612 INFO [AM.ZK.Worker-pool5-t6] master.RegionStates(1257): Transition {c71b20fd2578c13df17ccd07b9ab4dd5 state=OPENING, ts=1550183741580, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} to {c71b20fd2578c13df17ccd07b9ab4dd5 state=OPEN, ts=1550183741612, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:41,612 DEBUG [AM.ZK.Worker-pool5-t6] coordination.ZkOpenRegionCoordination(350): Handling OPENED of c71b20fd2578c13df17ccd07b9ab4dd5 from ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583; deleting unassigned node 2019-02-14 22:35:41,613 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/c71b20fd2578c13df17ccd07b9ab4dd5 2019-02-14 22:35:41,613 DEBUG [AM.ZK.Worker-pool5-t6] zookeeper.ZKAssign(479): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Deleted unassigned node c71b20fd2578c13df17ccd07b9ab4dd5 in expected state RS_ZK_REGION_OPENED 2019-02-14 22:35:41,614 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2019-02-14 22:35:41,615 DEBUG [AM.ZK.Worker-pool5-t8] master.AssignmentManager$5(1448): Znode hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5. deleted, state: {c71b20fd2578c13df17ccd07b9ab4dd5 state=OPEN, ts=1550183741612, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:41,615 DEBUG [AM.ZK.Worker-pool5-t8] master.RegionStates(553): Onlined c71b20fd2578c13df17ccd07b9ab4dd5 on ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:41,630 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2019-02-14 22:35:41,631 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2019-02-14 22:35:41,648 INFO [ip-172-31-47-35:40925.activeMasterManager] master.HMaster$13(2891): Client=null/null creating {NAME => 'default'} 2019-02-14 22:35:41,682 DEBUG [ip-172-31-47-35:40925.activeMasterManager] procedure2.ProcedureExecutor(769): Procedure CreateNamespaceProcedure (Namespace=default) id=2 owner=apurtell.hfs.0 state=RUNNABLE:CREATE_NAMESPACE_PREPARE added to the store. 2019-02-14 22:35:41,896 DEBUG [ProcedureExecutor-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/hbase:namespace/write-master:409250000000000 2019-02-14 22:35:41,896 DEBUG [ProcedureExecutor-0] procedure2.ProcedureExecutor(970): Procedure completed in 1.1950sec: CreateTableProcedure (table=hbase:namespace) id=1 owner=apurtell.hfs.0 state=FINISHED 2019-02-14 22:35:42,219 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2019-02-14 22:35:42,222 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node default with data: \x0A\x07default 2019-02-14 22:35:42,429 DEBUG [ProcedureExecutor-0] procedure2.ProcedureExecutor(970): Procedure completed in 676msec: CreateNamespaceProcedure (Namespace=default) id=2 owner=apurtell.hfs.0 state=FINISHED 2019-02-14 22:35:42,435 INFO [ip-172-31-47-35:40925.activeMasterManager] master.HMaster$13(2891): Client=null/null creating {NAME => 'hbase'} 2019-02-14 22:35:42,539 DEBUG [ip-172-31-47-35:40925.activeMasterManager] procedure2.ProcedureExecutor(769): Procedure CreateNamespaceProcedure (Namespace=hbase) id=3 owner=apurtell.hfs.0 state=RUNNABLE:CREATE_NAMESPACE_PREPARE added to the store. 2019-02-14 22:35:42,864 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2019-02-14 22:35:42,868 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node default with data: \x0A\x07default 2019-02-14 22:35:42,868 DEBUG [main-EventThread] hbase.ZKNamespaceManager(201): Updating namespace cache from node hbase with data: \x0A\x05hbase 2019-02-14 22:35:43,075 DEBUG [ProcedureExecutor-2] procedure2.ProcedureExecutor(970): Procedure completed in 535msec: CreateNamespaceProcedure (Namespace=hbase) id=3 owner=apurtell.hfs.0 state=FINISHED 2019-02-14 22:35:43,297 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.RecoverableZooKeeper(584): Node /hbase/namespace/default already exists 2019-02-14 22:35:43,298 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2019-02-14 22:35:43,300 DEBUG [ip-172-31-47-35:40925.activeMasterManager] zookeeper.RecoverableZooKeeper(584): Node /hbase/namespace/hbase already exists 2019-02-14 22:35:43,302 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2019-02-14 22:35:43,302 INFO [ip-172-31-47-35:40925.activeMasterManager] master.HMaster(895): Master has completed initialization 5.352sec 2019-02-14 22:35:43,302 DEBUG [ip-172-31-47-35:40925.activeMasterManager] procedure.MasterProcedureScheduler(387): Wake event ProcedureEvent(master initialized) 2019-02-14 22:35:43,304 INFO [ip-172-31-47-35:40925.activeMasterManager] quotas.MasterQuotaManager(63): Quota support disabled 2019-02-14 22:35:43,305 INFO [ip-172-31-47-35:40925.activeMasterManager] zookeeper.ZooKeeperWatcher(244): not a secure deployment, proceeding 2019-02-14 22:35:43,309 DEBUG [ip-172-31-47-35:40925.activeMasterManager] master.HMaster(947): Balancer post startup initialization complete, took 0 seconds 2019-02-14 22:35:43,327 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x6eaa21d8 connecting to ZooKeeper ensemble=localhost:50587 2019-02-14 22:35:43,331 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): hconnection-0x6eaa21d80x0, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2019-02-14 22:35:43,332 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(678): hconnection-0x6eaa21d8-0x168ee26a5f90006 connected 2019-02-14 22:35:43,333 DEBUG [main] ipc.AbstractRpcClient(199): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72e789cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2019-02-14 22:35:43,334 DEBUG [htable-pool13-t1] ipc.RpcConnection(126): Use SIMPLE authentication for service ClientService, sasl=false 2019-02-14 22:35:43,335 DEBUG [htable-pool13-t1] ipc.BlockingRpcConnection(431): Connecting to ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:39948 2019-02-14 22:35:43,336 DEBUG [RpcServer.listener,port=39948] ipc.RpcServer$Listener(939): RpcServer.listener,port=39948: connection from 172.31.47.35:51728; # active connections: 3 2019-02-14 22:35:43,336 INFO [RpcServer.reader=0,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Connection(1838): Auth successful for apurtell (auth:SIMPLE) 2019-02-14 22:35:43,337 INFO [RpcServer.reader=0,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Connection(1868): Connection from 172.31.47.35 port: 51728 with version info: version: "1.5.0" url: "git://ip-172-31-47-35.us-west-2.compute.internal/data/src/hbase" revision: "" user: "apurtell" date: "Thu Feb 14 22:35:08 UTC 2019" src_checksum: "60dfaeb22217478e0331d7332d9bc018" version_major: 1 version_minor: 5 2019-02-14 22:35:43,339 INFO [main] client.ConnectionManager$HConnectionImplementation(1774): Closing zookeeper sessionid=0x168ee26a5f90006 2019-02-14 22:35:43,341 DEBUG [main] ipc.AbstractRpcClient(483): Stopping rpc client 2019-02-14 22:35:43,343 DEBUG [RpcServer.reader=0,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Listener(971): RpcServer.listener,port=39948: DISCONNECTING client 172.31.47.35:51728 because read count=-1. Number of active connections: 3 2019-02-14 22:35:43,359 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x39e67516 connecting to ZooKeeper ensemble=localhost:50587 2019-02-14 22:35:43,363 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): hconnection-0x39e675160x0, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2019-02-14 22:35:43,363 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(678): hconnection-0x39e67516-0x168ee26a5f90007 connected 2019-02-14 22:35:43,363 DEBUG [main] ipc.AbstractRpcClient(199): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bb003e9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2019-02-14 22:35:43,364 INFO [main] hbase.HBaseTestingUtility(1125): Minicluster is up 2019-02-14 22:35:43,364 INFO [main] hbase.HBaseTestingUtility(1309): The hbase.fs.tmp.dir is set to /user/apurtell/hbase-staging 2019-02-14 22:35:43,398 INFO [main] hbase.ResourceChecker(147): before: regionserver.compactions.TestFIFOCompactionPolicy#testSanityCheckMinVersion Thread=226, OpenFileDescriptor=548, MaxFileDescriptor=65536, SystemLoadAverage=270, ProcessCount=429, AvailableMemoryMB=9169, ConnectionCount=0 2019-02-14 22:35:43,408 DEBUG [main] ipc.RpcConnection(126): Use SIMPLE authentication for service MasterService, sasl=false 2019-02-14 22:35:43,409 DEBUG [main] ipc.BlockingRpcConnection(431): Connecting to ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:40925 2019-02-14 22:35:43,410 DEBUG [RpcServer.listener,port=40925] ipc.RpcServer$Listener(939): RpcServer.listener,port=40925: connection from 172.31.47.35:42176; # active connections: 2 2019-02-14 22:35:43,411 INFO [RpcServer.reader=2,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=40925] ipc.RpcServer$Connection(1838): Auth successful for apurtell (auth:SIMPLE) 2019-02-14 22:35:43,411 INFO [RpcServer.reader=2,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=40925] ipc.RpcServer$Connection(1868): Connection from 172.31.47.35 port: 42176 with version info: version: "1.5.0" url: "git://ip-172-31-47-35.us-west-2.compute.internal/data/src/hbase" revision: "" user: "apurtell" date: "Thu Feb 14 22:35:08 UTC 2019" src_checksum: "60dfaeb22217478e0331d7332d9bc018" version_major: 1 version_minor: 5 2019-02-14 22:35:43,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] ipc.CallRunner(130): RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925: callId: 1 service: MasterService methodName: CreateTable size: 578 connection: 172.31.47.35:42176 deadline: 1550183803416 org.apache.hadoop.hbase.DoNotRetryIOException: MIN_VERSION > 0 is not supported for FIFO compaction Set hbase.table.sanity.checks to false at conf or table descriptor if you want to bypass sanity checks at org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1993) at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1859) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1741) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:482) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:63400) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) Caused by: java.io.IOException: MIN_VERSION > 0 is not supported for FIFO compaction at org.apache.hadoop.hbase.master.HMaster.checkCompactionPolicy(HMaster.java:1972) at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1857) ... 7 more 2019-02-14 22:35:43,456 INFO [main] hbase.ResourceChecker(171): after: regionserver.compactions.TestFIFOCompactionPolicy#testSanityCheckMinVersion Thread=227 (was 226) Potentially hanging thread: IPC Client (2138485092) connection to ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:40925 from apurtell java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.BlockingRpcConnection.waitForWork(BlockingRpcConnection.java:322) org.apache.hadoop.hbase.ipc.BlockingRpcConnection.run(BlockingRpcConnection.java:333) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=550 (was 548) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=65536 (was 65536), SystemLoadAverage=270 (was 270), ProcessCount=429 (was 429), AvailableMemoryMB=9165 (was 9169), ConnectionCount=0 (was 0) 2019-02-14 22:35:43,476 INFO [main] hbase.ResourceChecker(147): before: regionserver.compactions.TestFIFOCompactionPolicy#testSanityCheckTTL Thread=226, OpenFileDescriptor=546, MaxFileDescriptor=65536, SystemLoadAverage=270, ProcessCount=429, AvailableMemoryMB=9159, ConnectionCount=0 2019-02-14 22:35:43,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] ipc.CallRunner(130): RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925: callId: 3 service: MasterService methodName: CreateTable size: 580 connection: 172.31.47.35:42176 deadline: 1550183803479 org.apache.hadoop.hbase.DoNotRetryIOException: Default TTL is not supported for FIFO compaction Set hbase.table.sanity.checks to false at conf or table descriptor if you want to bypass sanity checks at org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1993) at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1859) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1741) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:482) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:63400) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) Caused by: java.io.IOException: Default TTL is not supported for FIFO compaction at org.apache.hadoop.hbase.master.HMaster.checkCompactionPolicy(HMaster.java:1966) at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1857) ... 7 more 2019-02-14 22:35:43,498 INFO [main] hbase.ResourceChecker(171): after: regionserver.compactions.TestFIFOCompactionPolicy#testSanityCheckTTL Thread=226 (was 226), OpenFileDescriptor=546 (was 546), MaxFileDescriptor=65536 (was 65536), SystemLoadAverage=270 (was 270), ProcessCount=429 (was 429), AvailableMemoryMB=9157 (was 9159), ConnectionCount=0 (was 0) 2019-02-14 22:35:43,518 INFO [main] hbase.ResourceChecker(147): before: regionserver.compactions.TestFIFOCompactionPolicy#testFIFOCompactionPolicyExpiredEmptyHFiles Thread=226, OpenFileDescriptor=546, MaxFileDescriptor=65536, SystemLoadAverage=270, ProcessCount=429, AvailableMemoryMB=9156, ConnectionCount=0 2019-02-14 22:35:43,522 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.HMaster$1(1749): Client=apurtell//172.31.47.35 create 'testFIFOCompactionPolicyExpiredEmptyHFiles', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hstore.defaultengine.compactionpolicy.class' => 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy', 'hbase.regionserver.region.split.policy' => 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy'}}, {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => '1 SECOND', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2019-02-14 22:35:43,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure2.ProcedureExecutor(769): Procedure CreateTableProcedure (table=testFIFOCompactionPolicyExpiredEmptyHFiles) id=4 owner=apurtell state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2019-02-14 22:35:43,634 DEBUG [ProcedureExecutor-3] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/testFIFOCompactionPolicyExpiredEmptyHFiles/write-master:409250000000000 2019-02-14 22:35:43,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(1059): Checking to see if procedure is done procId=4 2019-02-14 22:35:43,638 DEBUG [ProcedureExecutor-3] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/table/testFIFOCompactionPolicyExpiredEmptyHFiles because node does not exist (not an error) 2019-02-14 22:35:43,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(1059): Checking to see if procedure is done procId=4 2019-02-14 22:35:43,757 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741834_1010{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 540 2019-02-14 22:35:43,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(1059): Checking to see if procedure is done procId=4 2019-02-14 22:35:44,161 DEBUG [ProcedureExecutor-3] util.FSTableDescriptors(660): Wrote descriptor into: hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/.tmp/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/.tabledesc/.tableinfo.0000000001 2019-02-14 22:35:44,163 INFO [RegionOpenAndInitThread-testFIFOCompactionPolicyExpiredEmptyHFiles-1] regionserver.HRegion(6831): creating HRegion testFIFOCompactionPolicyExpiredEmptyHFiles HTD == 'testFIFOCompactionPolicyExpiredEmptyHFiles', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hstore.defaultengine.compactionpolicy.class' => 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy', 'hbase.regionserver.region.split.policy' => 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy'}}, {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => '1 SECOND', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/.tmp Table name == testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:44,177 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741835_1011{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f3ca40dc-91f2-42ec-82fe-8f7a3a14736f:NORMAL:127.0.0.1:38317|RBW]]} size 77 2019-02-14 22:35:44,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(1059): Checking to see if procedure is done procId=4 2019-02-14 22:35:44,579 DEBUG [RegionOpenAndInitThread-testFIFOCompactionPolicyExpiredEmptyHFiles-1] regionserver.HRegion(826): Instantiated testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:44,579 DEBUG [RegionOpenAndInitThread-testFIFOCompactionPolicyExpiredEmptyHFiles-1] regionserver.HRegion(1526): Closing testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0.: disabling compactions & flushes 2019-02-14 22:35:44,579 DEBUG [RegionOpenAndInitThread-testFIFOCompactionPolicyExpiredEmptyHFiles-1] regionserver.HRegion(1553): Updates disabled for region testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:44,579 INFO [RegionOpenAndInitThread-testFIFOCompactionPolicyExpiredEmptyHFiles-1] regionserver.HRegion(1660): Closed testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:44,690 INFO [ProcedureExecutor-3] hbase.MetaTableAccessor(1234): Added 1 2019-02-14 22:35:44,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(1059): Checking to see if procedure is done procId=4 2019-02-14 22:35:44,794 INFO [ProcedureExecutor-3] zookeeper.ZKTableStateManager(100): Moving table testFIFOCompactionPolicyExpiredEmptyHFiles state from null to ENABLING 2019-02-14 22:35:44,799 INFO [ProcedureExecutor-3] master.AssignmentManager(1724): Assigning 1 region(s) to ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:44,799 DEBUG [ProcedureExecutor-3] zookeeper.ZKAssign(174): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Async create of unassigned node 39958fd836f676bdcfb70cc61f4ef4c0 with OFFLINE state 2019-02-14 22:35:44,801 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2019-02-14 22:35:44,801 DEBUG [main-EventThread] master.OfflineCallback(69): rs={39958fd836f676bdcfb70cc61f4ef4c0 state=OFFLINE, ts=1550183744799, server=null}, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:44,801 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={39958fd836f676bdcfb70cc61f4ef4c0 state=OFFLINE, ts=1550183744799, server=null}, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:44,804 DEBUG [ProcedureExecutor-3] master.AssignmentManager(1777): ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 unassigned znodes=1 of total=1; oldCounter=0 2019-02-14 22:35:44,804 INFO [ProcedureExecutor-3] master.RegionStates(1257): Transition {39958fd836f676bdcfb70cc61f4ef4c0 state=OFFLINE, ts=1550183744799, server=null} to {39958fd836f676bdcfb70cc61f4ef4c0 state=PENDING_OPEN, ts=1550183744804, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:44,805 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.RSRpcServices(1840): Open testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:44,811 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-0] zookeeper.ZKAssign(805): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioning 39958fd836f676bdcfb70cc61f4ef4c0 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2019-02-14 22:35:44,813 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/39958fd836f676bdcfb70cc61f4ef4c0 2019-02-14 22:35:44,813 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-0] zookeeper.ZKAssign(880): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioned node 39958fd836f676bdcfb70cc61f4ef4c0 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2019-02-14 22:35:44,813 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-0] regionserver.HRegion(7020): Opening region: {ENCODED => 39958fd836f676bdcfb70cc61f4ef4c0, NAME => 'testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0.', STARTKEY => '', ENDKEY => ''} 2019-02-14 22:35:44,814 DEBUG [AM.ZK.Worker-pool5-t10] master.AssignmentManager(1052): Handling RS_ZK_REGION_OPENING, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985, region=39958fd836f676bdcfb70cc61f4ef4c0, current_state={39958fd836f676bdcfb70cc61f4ef4c0 state=PENDING_OPEN, ts=1550183744804, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:44,814 INFO [AM.ZK.Worker-pool5-t10] master.RegionStates(1257): Transition {39958fd836f676bdcfb70cc61f4ef4c0 state=PENDING_OPEN, ts=1550183744804, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} to {39958fd836f676bdcfb70cc61f4ef4c0 state=OPENING, ts=1550183744814, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:44,814 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table testFIFOCompactionPolicyExpiredEmptyHFiles 39958fd836f676bdcfb70cc61f4ef4c0 2019-02-14 22:35:44,814 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-0] regionserver.HRegion(826): Instantiated testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:44,818 DEBUG [StoreOpener-39958fd836f676bdcfb70cc61f4ef4c0-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f 2019-02-14 22:35:44,819 DEBUG [StoreOpener-39958fd836f676bdcfb70cc61f4ef4c0-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f 2019-02-14 22:35:44,819 INFO [StoreOpener-39958fd836f676bdcfb70cc61f4ef4c0-1] hfile.CacheConfig(242): Created cacheConfig for f: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2019-02-14 22:35:44,820 INFO [StoreOpener-39958fd836f676bdcfb70cc61f4ef4c0-1] compactions.CompactionConfiguration(149): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2019-02-14 22:35:44,821 INFO [StoreOpener-39958fd836f676bdcfb70cc61f4ef4c0-1] regionserver.HStore(370): Store=f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, encoding=NONE, compression=NONE 2019-02-14 22:35:44,823 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-0] regionserver.HRegion(4352): Found 0 recovered edits file(s) under hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0 2019-02-14 22:35:44,824 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-0] regionserver.HRegion(4352): Found 0 recovered edits file(s) under hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0 2019-02-14 22:35:44,828 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2019-02-14 22:35:44,832 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-0] wal.WALSplitter(751): Wrote region seqId=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2019-02-14 22:35:44,832 INFO [RS_OPEN_REGION-ip-172-31-47-35:39948-0] regionserver.HRegion(977): Onlined 39958fd836f676bdcfb70cc61f4ef4c0; next sequenceid=2 2019-02-14 22:35:44,833 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-0] zookeeper.ZKAssign(643): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Attempting to retransition opening state of node 39958fd836f676bdcfb70cc61f4ef4c0 2019-02-14 22:35:44,834 INFO [PostOpenDeployTasks:39958fd836f676bdcfb70cc61f4ef4c0] regionserver.HRegionServer(2058): Post open deploy tasks for testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:44,836 INFO [PostOpenDeployTasks:39958fd836f676bdcfb70cc61f4ef4c0] hbase.MetaTableAccessor(1434): Updated row testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. with server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:44,836 DEBUG [PostOpenDeployTasks:39958fd836f676bdcfb70cc61f4ef4c0] regionserver.HRegionServer(2092): Finished post open deploy task for testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:44,837 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-0] zookeeper.ZKAssign(805): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioning 39958fd836f676bdcfb70cc61f4ef4c0 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2019-02-14 22:35:44,839 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/39958fd836f676bdcfb70cc61f4ef4c0 2019-02-14 22:35:44,839 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-0] zookeeper.ZKAssign(880): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioned node 39958fd836f676bdcfb70cc61f4ef4c0 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2019-02-14 22:35:44,839 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-0] coordination.ZkOpenRegionCoordination(91): Transitioned 39958fd836f676bdcfb70cc61f4ef4c0 to OPENED in zk on ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:44,839 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-0] handler.OpenRegionHandler(172): Opened testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. on ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:44,840 DEBUG [AM.ZK.Worker-pool5-t11] master.AssignmentManager(1052): Handling RS_ZK_REGION_OPENED, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985, region=39958fd836f676bdcfb70cc61f4ef4c0, current_state={39958fd836f676bdcfb70cc61f4ef4c0 state=OPENING, ts=1550183744814, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:44,840 INFO [AM.ZK.Worker-pool5-t11] master.RegionStates(1257): Transition {39958fd836f676bdcfb70cc61f4ef4c0 state=OPENING, ts=1550183744814, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} to {39958fd836f676bdcfb70cc61f4ef4c0 state=OPEN, ts=1550183744840, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:44,840 DEBUG [AM.ZK.Worker-pool5-t11] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 39958fd836f676bdcfb70cc61f4ef4c0 from ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583; deleting unassigned node 2019-02-14 22:35:44,841 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/39958fd836f676bdcfb70cc61f4ef4c0 2019-02-14 22:35:44,842 DEBUG [AM.ZK.Worker-pool5-t11] zookeeper.ZKAssign(479): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Deleted unassigned node 39958fd836f676bdcfb70cc61f4ef4c0 in expected state RS_ZK_REGION_OPENED 2019-02-14 22:35:44,842 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2019-02-14 22:35:44,842 DEBUG [AM.ZK.Worker-pool5-t12] master.AssignmentManager$5(1448): Znode testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. deleted, state: {39958fd836f676bdcfb70cc61f4ef4c0 state=OPEN, ts=1550183744840, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:44,842 DEBUG [AM.ZK.Worker-pool5-t12] master.RegionStates(553): Onlined 39958fd836f676bdcfb70cc61f4ef4c0 on ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:44,842 DEBUG [ProcedureExecutor-3] master.AssignmentManager(1919): Bulk assigning done for ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:44,842 INFO [ProcedureExecutor-3] zookeeper.ZKTableStateManager(100): Moving table testFIFOCompactionPolicyExpiredEmptyHFiles state from ENABLING to ENABLED 2019-02-14 22:35:45,162 DEBUG [ProcedureExecutor-3] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/testFIFOCompactionPolicyExpiredEmptyHFiles/write-master:409250000000000 2019-02-14 22:35:45,163 DEBUG [ProcedureExecutor-3] procedure2.ProcedureExecutor(970): Procedure completed in 1.5340sec: CreateTableProcedure (table=testFIFOCompactionPolicyExpiredEmptyHFiles) id=4 owner=apurtell state=FINISHED 2019-02-14 22:35:45,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(1059): Checking to see if procedure is done procId=4 2019-02-14 22:35:45,754 DEBUG [htable-pool14-t1] ipc.RpcConnection(126): Use SIMPLE authentication for service ClientService, sasl=false 2019-02-14 22:35:45,754 DEBUG [htable-pool14-t1] ipc.BlockingRpcConnection(431): Connecting to ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:39948 2019-02-14 22:35:45,755 DEBUG [RpcServer.listener,port=39948] ipc.RpcServer$Listener(939): RpcServer.listener,port=39948: connection from 172.31.47.35:51788; # active connections: 3 2019-02-14 22:35:45,755 INFO [RpcServer.reader=1,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Connection(1838): Auth successful for apurtell (auth:SIMPLE) 2019-02-14 22:35:45,755 INFO [RpcServer.reader=1,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Connection(1868): Connection from 172.31.47.35 port: 51788 with version info: version: "1.5.0" url: "git://ip-172-31-47-35.us-west-2.compute.internal/data/src/hbase" revision: "" user: "apurtell" date: "Thu Feb 14 22:35:08 UTC 2019" src_checksum: "60dfaeb22217478e0331d7332d9bc018" version_major: 1 version_minor: 5 2019-02-14 22:35:45,761 INFO [main] client.HBaseAdmin$CreateTableFuture(795): Created testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:45,762 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x365a6a43 connecting to ZooKeeper ensemble=localhost:50587 2019-02-14 22:35:45,766 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): hconnection-0x365a6a430x0, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2019-02-14 22:35:45,766 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(678): hconnection-0x365a6a43-0x168ee26a5f90008 connected 2019-02-14 22:35:45,767 DEBUG [main] ipc.AbstractRpcClient(199): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@99407c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2019-02-14 22:35:45,769 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2019-02-14 22:35:45,770 DEBUG [htable-pool16-t1] ipc.RpcConnection(126): Use SIMPLE authentication for service ClientService, sasl=false 2019-02-14 22:35:45,770 DEBUG [htable-pool16-t1] ipc.BlockingRpcConnection(431): Connecting to ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:39948 2019-02-14 22:35:45,771 DEBUG [RpcServer.listener,port=39948] ipc.RpcServer$Listener(939): RpcServer.listener,port=39948: connection from 172.31.47.35:51792; # active connections: 4 2019-02-14 22:35:45,771 INFO [RpcServer.reader=2,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Connection(1838): Auth successful for apurtell (auth:SIMPLE) 2019-02-14 22:35:45,771 INFO [RpcServer.reader=2,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Connection(1868): Connection from 172.31.47.35 port: 51792 with version info: version: "1.5.0" url: "git://ip-172-31-47-35.us-west-2.compute.internal/data/src/hbase" revision: "" user: "apurtell" date: "Thu Feb 14 22:35:08 UTC 2019" src_checksum: "60dfaeb22217478e0331d7332d9bc018" version_major: 1 version_minor: 5 2019-02-14 22:35:45,774 INFO [main] client.ConnectionManager$HConnectionImplementation(1774): Closing zookeeper sessionid=0x168ee26a5f90008 2019-02-14 22:35:45,776 DEBUG [main] ipc.AbstractRpcClient(483): Stopping rpc client 2019-02-14 22:35:45,777 DEBUG [RpcServer.reader=2,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Listener(971): RpcServer.listener,port=39948: DISCONNECTING client 172.31.47.35:51792 because read count=-1. Number of active connections: 4 2019-02-14 22:35:45,778 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2019-02-14 22:35:45,778 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x1cb7936c connecting to ZooKeeper ensemble=localhost:50587 2019-02-14 22:35:45,787 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): hconnection-0x1cb7936c0x0, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2019-02-14 22:35:45,788 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(678): hconnection-0x1cb7936c-0x168ee26a5f90009 connected 2019-02-14 22:35:45,788 DEBUG [main] ipc.AbstractRpcClient(199): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@128c502c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2019-02-14 22:35:45,790 DEBUG [hconnection-0x1cb7936c-metaLookup-shared--pool18-t1] ipc.RpcConnection(126): Use SIMPLE authentication for service ClientService, sasl=false 2019-02-14 22:35:45,790 DEBUG [hconnection-0x1cb7936c-metaLookup-shared--pool18-t1] ipc.BlockingRpcConnection(431): Connecting to ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:39948 2019-02-14 22:35:45,790 DEBUG [RpcServer.listener,port=39948] ipc.RpcServer$Listener(939): RpcServer.listener,port=39948: connection from 172.31.47.35:51796; # active connections: 4 2019-02-14 22:35:45,791 INFO [RpcServer.reader=0,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Connection(1838): Auth successful for apurtell (auth:SIMPLE) 2019-02-14 22:35:45,791 INFO [RpcServer.reader=0,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Connection(1868): Connection from 172.31.47.35 port: 51796 with version info: version: "1.5.0" url: "git://ip-172-31-47-35.us-west-2.compute.internal/data/src/hbase" revision: "" user: "apurtell" date: "Thu Feb 14 22:35:08 UTC 2019" src_checksum: "60dfaeb22217478e0331d7332d9bc018" version_major: 1 version_minor: 5 2019-02-14 22:35:45,811 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(753): Client=apurtell//172.31.47.35 procedure request for: flush-table-proc 2019-02-14 22:35:45,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(177): Submitting procedure testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:45,819 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' 2019-02-14 22:35:45,819 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:45,820 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'acquire' 2019-02-14 22:35:45,820 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles', kicking off acquire phase on members. 2019-02-14 22:35:45,820 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:45,820 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:45,822 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:45,822 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:45,823 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:45,823 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:45,823 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:45,823 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2019-02-14 22:35:45,823 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:45,824 DEBUG [main-EventThread] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:45,824 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2019-02-14 22:35:45,824 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:45,824 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:45,826 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:45,827 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' with timeout 60000ms 2019-02-14 22:35:45,827 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2019-02-14 22:35:45,827 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:45,828 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'acquire' stage 2019-02-14 22:35:45,828 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2019-02-14 22:35:45,828 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2019-02-14 22:35:45,828 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool5-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:45,830 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool5-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. started... 2019-02-14 22:35:45,832 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool5-thread-1] regionserver.HRegion(2432): Flushing 1/1 column families, memstore=120 B 2019-02-14 22:35:45,845 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool5-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f 2019-02-14 22:35:45,845 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool5-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f 2019-02-14 22:35:45,886 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2019-02-14 22:35:45,886 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2019-02-14 22:35:45,886 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2019-02-14 22:35:45,918 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741836_1012{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 4527 2019-02-14 22:35:46,320 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool5-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=6, memsize=120, hasBloomFilter=true, into tmp file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f/374b7b8dc7314f24a42098a19473b3bc 2019-02-14 22:35:46,345 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool5-thread-1] regionserver.StoreFile(495): HFile Bloom filter type for 374b7b8dc7314f24a42098a19473b3bc: NONE, but ROW specified in column family configuration 2019-02-14 22:35:46,362 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool5-thread-1] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f/374b7b8dc7314f24a42098a19473b3bc as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/374b7b8dc7314f24a42098a19473b3bc 2019-02-14 22:35:46,371 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool5-thread-1] regionserver.StoreFile(495): HFile Bloom filter type for 374b7b8dc7314f24a42098a19473b3bc: NONE, but ROW specified in column family configuration 2019-02-14 22:35:46,371 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool5-thread-1] regionserver.HStore(1050): Added hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/374b7b8dc7314f24a42098a19473b3bc, entries=0, sequenceid=6, filesize=4.4 K 2019-02-14 22:35:46,373 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool5-thread-1] regionserver.HRegion(2732): Finished memstore flush of ~120 B/120, currentsize=0 B/0 for region testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. in 541ms, sequenceid=6, compaction requested=true 2019-02-14 22:35:46,374 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool5-thread-1] regionserver.HRegion(2233): Flush status journal: Acquiring readlock on region at 1550183745830 Running coprocessor pre-flush hooks at 1550183745831 Obtaining lock to block concurrent updates at 1550183745832 Preparing to flush by snapshotting stores in 39958fd836f676bdcfb70cc61f4ef4c0 at 1550183745832 Finished memstore snapshotting testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0., syncing WAL and waiting on mvcc, flushsize=120 at 1550183745838 Flushing stores of testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. at 1550183745839 Flushing f: creating writer at 1550183745841 Flushing f: appending metadata at 1550183745899 Flushing f: closing flushed file at 1550183745899 Flushing f: reopening flushed file at 1550183746363 Finished memstore flush of ~120 B/120, currentsize=0 B/0 for region testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. in 541ms, sequenceid=6, compaction requested=true at 1550183746373 Running post-flush coprocessor hooks at 1550183746374 Flush successful at 1550183746374 2019-02-14 22:35:46,374 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool5-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:46,374 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2019-02-14 22:35:46,375 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2019-02-14 22:35:46,375 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,375 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' locally acquired 2019-02-14 22:35:46,375 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure (testFIFOCompactionPolicyExpiredEmptyHFiles) in zk 2019-02-14 22:35:46,377 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,377 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,378 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,378 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:46,378 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:46,378 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,378 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2019-02-14 22:35:46,378 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:46,378 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:46,379 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,379 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,379 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:46,379 DEBUG [main-EventThread] procedure.Procedure(298): member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' on coordinator 2019-02-14 22:35:46,379 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'in-barrier' execution. 2019-02-14 22:35:46,379 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@443433b6[Count = 0] remaining members to acquire global barrier 2019-02-14 22:35:46,379 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,381 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,381 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,381 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,382 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,382 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' received 'reached' from coordinator. 2019-02-14 22:35:46,382 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2019-02-14 22:35:46,382 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' locally completed 2019-02-14 22:35:46,382 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' completed for member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' in zk 2019-02-14 22:35:46,384 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,384 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' has notified controller of completion 2019-02-14 22:35:46,384 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:46,384 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,384 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:46,384 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:46,384 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' completed. 2019-02-14 22:35:46,385 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:46,385 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:46,385 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,385 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,386 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:46,386 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,386 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,387 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985': 2019-02-14 22:35:46,387 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' released barrier for procedure'testFIFOCompactionPolicyExpiredEmptyHFiles', counting down latch. Waiting for 0 more 2019-02-14 22:35:46,387 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' execution completed 2019-02-14 22:35:46,387 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2019-02-14 22:35:46,387 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2019-02-14 22:35:46,387 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,387 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure testFIFOCompactionPolicyExpiredEmptyHFilesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2019-02-14 22:35:46,392 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,392 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,392 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,392 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,392 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,392 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:46,392 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:46,393 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:46,393 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,393 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:46,393 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:46,393 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:46,393 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,393 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,393 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:46,394 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,394 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,394 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,394 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:46,394 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,395 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,400 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,400 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,400 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:46,400 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:46,400 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:46,400 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,400 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,401 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,401 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,401 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,401 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:46,401 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'testFIFOCompactionPolicyExpiredEmptyHFiles' 2019-02-14 22:35:46,401 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2019-02-14 22:35:46,401 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:46,401 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:46,402 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:46,402 DEBUG [main] client.HBaseAdmin(4115): Waiting a max of 300000 ms for procedure 'flush-table-proc : testFIFOCompactionPolicyExpiredEmptyHFiles'' to complete. (max 8571 ms per retry) 2019-02-14 22:35:46,403 DEBUG [main] client.HBaseAdmin(4124): (#1) Sleeping: 100ms while waiting for procedure completion. 2019-02-14 22:35:46,503 DEBUG [main] client.HBaseAdmin(4130): Getting current status of procedure from master... 2019-02-14 22:35:46,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(989): Checking to see if procedure from request:flush-table-proc is done 2019-02-14 22:35:46,513 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(753): Client=apurtell//172.31.47.35 procedure request for: flush-table-proc 2019-02-14 22:35:46,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(155): Procedure testFIFOCompactionPolicyExpiredEmptyHFiles was in running list but was completed. Accepting new attempt. 2019-02-14 22:35:46,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(177): Submitting procedure testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,515 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' 2019-02-14 22:35:46,515 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:46,515 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'acquire' 2019-02-14 22:35:46,515 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles', kicking off acquire phase on members. 2019-02-14 22:35:46,516 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,516 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,517 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:46,517 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,517 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:46,517 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:46,517 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,517 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2019-02-14 22:35:46,518 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,518 DEBUG [main-EventThread] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,518 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2019-02-14 22:35:46,518 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,518 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,519 WARN [main-EventThread] procedure.ProcedureMember(142): A completed old subproc testFIFOCompactionPolicyExpiredEmptyHFiles is still present, removing 2019-02-14 22:35:46,519 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,519 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' with timeout 60000ms 2019-02-14 22:35:46,519 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:46,519 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'acquire' stage 2019-02-14 22:35:46,519 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2019-02-14 22:35:46,519 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2019-02-14 22:35:46,520 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool6-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:46,520 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool6-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. started... 2019-02-14 22:35:46,520 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool6-thread-1] regionserver.HRegion(2432): Flushing 1/1 column families, memstore=120 B 2019-02-14 22:35:46,522 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool6-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f 2019-02-14 22:35:46,522 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool6-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f 2019-02-14 22:35:46,534 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741837_1013{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 4527 2019-02-14 22:35:46,936 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool6-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=11, memsize=120, hasBloomFilter=true, into tmp file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f/2456003b842340a8a586d3721c8cc5e1 2019-02-14 22:35:46,943 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool6-thread-1] regionserver.StoreFile(495): HFile Bloom filter type for 2456003b842340a8a586d3721c8cc5e1: NONE, but ROW specified in column family configuration 2019-02-14 22:35:46,945 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool6-thread-1] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f/2456003b842340a8a586d3721c8cc5e1 as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/2456003b842340a8a586d3721c8cc5e1 2019-02-14 22:35:46,953 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool6-thread-1] regionserver.StoreFile(495): HFile Bloom filter type for 2456003b842340a8a586d3721c8cc5e1: NONE, but ROW specified in column family configuration 2019-02-14 22:35:46,954 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool6-thread-1] regionserver.HStore(1050): Added hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/2456003b842340a8a586d3721c8cc5e1, entries=0, sequenceid=11, filesize=4.4 K 2019-02-14 22:35:46,955 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool6-thread-1] regionserver.HRegion(2732): Finished memstore flush of ~120 B/120, currentsize=0 B/0 for region testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. in 435ms, sequenceid=11, compaction requested=true 2019-02-14 22:35:46,955 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool6-thread-1] regionserver.HRegion(2233): Flush status journal: Acquiring readlock on region at 1550183746520 Running coprocessor pre-flush hooks at 1550183746520 Obtaining lock to block concurrent updates at 1550183746520 Preparing to flush by snapshotting stores in 39958fd836f676bdcfb70cc61f4ef4c0 at 1550183746520 Finished memstore snapshotting testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0., syncing WAL and waiting on mvcc, flushsize=120 at 1550183746520 Flushing stores of testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. at 1550183746521 Flushing f: creating writer at 1550183746521 Flushing f: appending metadata at 1550183746525 Flushing f: closing flushed file at 1550183746525 Flushing f: reopening flushed file at 1550183746945 Finished memstore flush of ~120 B/120, currentsize=0 B/0 for region testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. in 435ms, sequenceid=11, compaction requested=true at 1550183746955 Running post-flush coprocessor hooks at 1550183746955 Flush successful at 1550183746955 2019-02-14 22:35:46,955 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool6-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:46,955 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2019-02-14 22:35:46,956 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2019-02-14 22:35:46,956 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,956 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' locally acquired 2019-02-14 22:35:46,956 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure (testFIFOCompactionPolicyExpiredEmptyHFiles) in zk 2019-02-14 22:35:46,958 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,958 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,958 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,959 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:46,959 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:46,959 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,959 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2019-02-14 22:35:46,959 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:46,959 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:46,959 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,960 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,960 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:46,960 DEBUG [main-EventThread] procedure.Procedure(298): member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' on coordinator 2019-02-14 22:35:46,960 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@24f82428[Count = 0] remaining members to acquire global barrier 2019-02-14 22:35:46,960 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'in-barrier' execution. 2019-02-14 22:35:46,960 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,962 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,962 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,962 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,962 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' received 'reached' from coordinator. 2019-02-14 22:35:46,962 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' locally completed 2019-02-14 22:35:46,962 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' completed for member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' in zk 2019-02-14 22:35:46,962 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,962 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2019-02-14 22:35:46,964 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,964 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' has notified controller of completion 2019-02-14 22:35:46,964 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:46,964 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' completed. 2019-02-14 22:35:46,964 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,965 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:46,965 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:46,965 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:46,965 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:46,966 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,966 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,966 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:46,966 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,966 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,967 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985': 2019-02-14 22:35:46,967 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' released barrier for procedure'testFIFOCompactionPolicyExpiredEmptyHFiles', counting down latch. Waiting for 0 more 2019-02-14 22:35:46,967 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' execution completed 2019-02-14 22:35:46,967 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2019-02-14 22:35:46,967 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2019-02-14 22:35:46,967 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,967 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure testFIFOCompactionPolicyExpiredEmptyHFilesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2019-02-14 22:35:46,969 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,969 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,969 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:46,969 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:46,969 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,969 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,969 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,969 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:46,969 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,970 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:46,970 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:46,970 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:46,970 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,970 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,970 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:46,970 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,970 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,971 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,971 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:46,971 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,971 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,974 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:46,974 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:46,974 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:46,974 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:46,974 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'testFIFOCompactionPolicyExpiredEmptyHFiles' 2019-02-14 22:35:46,975 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2019-02-14 22:35:46,975 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:46,975 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:46,975 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:46,975 DEBUG [main-EventThread] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles because node does not exist (not an error) 2019-02-14 22:35:46,975 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,975 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,975 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,975 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:46,975 DEBUG [main] client.HBaseAdmin(4115): Waiting a max of 300000 ms for procedure 'flush-table-proc : testFIFOCompactionPolicyExpiredEmptyHFiles'' to complete. (max 8571 ms per retry) 2019-02-14 22:35:46,975 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,976 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,976 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:46,976 DEBUG [main] client.HBaseAdmin(4124): (#1) Sleeping: 100ms while waiting for procedure completion. 2019-02-14 22:35:47,076 DEBUG [main] client.HBaseAdmin(4130): Getting current status of procedure from master... 2019-02-14 22:35:47,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(989): Checking to see if procedure from request:flush-table-proc is done 2019-02-14 22:35:47,081 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hbase-admin-on-hconnection-0x39e67516 connecting to ZooKeeper ensemble=localhost:50587 2019-02-14 22:35:47,084 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): hbase-admin-on-hconnection-0x39e675160x0, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2019-02-14 22:35:47,085 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(678): hbase-admin-on-hconnection-0x39e67516-0x168ee26a5f9000a connected 2019-02-14 22:35:47,085 DEBUG [main] ipc.RpcConnection(126): Use SIMPLE authentication for service AdminService, sasl=false 2019-02-14 22:35:47,085 DEBUG [main] ipc.BlockingRpcConnection(431): Connecting to ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:39948 2019-02-14 22:35:47,086 DEBUG [RpcServer.listener,port=39948] ipc.RpcServer$Listener(939): RpcServer.listener,port=39948: connection from 172.31.47.35:51830; # active connections: 5 2019-02-14 22:35:47,086 INFO [RpcServer.reader=1,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Connection(1838): Auth successful for apurtell (auth:SIMPLE) 2019-02-14 22:35:47,086 INFO [RpcServer.reader=1,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Connection(1868): Connection from 172.31.47.35 port: 51830 with version info: version: "1.5.0" url: "git://ip-172-31-47-35.us-west-2.compute.internal/data/src/hbase" revision: "" user: "apurtell" date: "Thu Feb 14 22:35:08 UTC 2019" src_checksum: "60dfaeb22217478e0331d7332d9bc018" version_major: 1 version_minor: 5 2019-02-14 22:35:47,086 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.RSRpcServices(1480): Compacting testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:47,087 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.DefaultStoreFileManager(204): Found an expired store file: hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/374b7b8dc7314f24a42098a19473b3bc whose maxTimeStamp is 1550183735788, which is below 1550183746087 2019-02-14 22:35:47,091 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.HStore(1938): Completing compaction... 2019-02-14 22:35:47,091 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.HStore(1857): Completed removal of 1 unnecessary (expired) file(s) in f of testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0.; total size for store is 4.4 K 2019-02-14 22:35:47,092 WARN [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] compactions.FIFOCompactionPolicy(63): Major compaction is not supported for FIFO compaction policy. Ignore the flag. 2019-02-14 22:35:47,093 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.HStore(1812): 39958fd836f676bdcfb70cc61f4ef4c0 - f: Initiating minor compaction 2019-02-14 22:35:47,094 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.CompactSplitThread(400): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@1b72eea9; Because: User-triggered major compaction; compaction_queue=(0:0), split_queue=0, merge_queue=0 2019-02-14 22:35:47,094 INFO [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.HRegion(2097): Starting compaction on f in region testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:47,095 INFO [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.HStore(1355): Starting compaction of 1 file(s) in f of testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. into tmpdir=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp, totalSize=4.4 K 2019-02-14 22:35:47,096 DEBUG [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] compactions.Compactor(197): Compacting hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/2456003b842340a8a586d3721c8cc5e1, keycount=0, bloomtype=NONE, size=4.4 K, encoding=NONE, compression=NONE, seqNum=11 2019-02-14 22:35:47,106 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(753): Client=apurtell//172.31.47.35 procedure request for: flush-table-proc 2019-02-14 22:35:47,106 INFO [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.StoreFile(495): HFile Bloom filter type for 2456003b842340a8a586d3721c8cc5e1: NONE, but ROW specified in column family configuration 2019-02-14 22:35:47,112 DEBUG [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f 2019-02-14 22:35:47,112 DEBUG [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f 2019-02-14 22:35:47,116 INFO [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] throttle.PressureAwareThroughputController(153): testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0.#f#compaction#2 average throughput is NaN MB/sec, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/sec 2019-02-14 22:35:47,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(155): Procedure testFIFOCompactionPolicyExpiredEmptyHFiles was in running list but was completed. Accepting new attempt. 2019-02-14 22:35:47,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(177): Submitting procedure testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,117 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' 2019-02-14 22:35:47,117 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:47,130 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'acquire' 2019-02-14 22:35:47,130 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles', kicking off acquire phase on members. 2019-02-14 22:35:47,131 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,131 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,132 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:47,132 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,132 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:47,133 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:47,133 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,133 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2019-02-14 22:35:47,133 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,133 DEBUG [main-EventThread] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,134 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2019-02-14 22:35:47,134 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,134 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,134 WARN [main-EventThread] procedure.ProcedureMember(142): A completed old subproc testFIFOCompactionPolicyExpiredEmptyHFiles is still present, removing 2019-02-14 22:35:47,135 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,135 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' with timeout 60000ms 2019-02-14 22:35:47,135 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:47,147 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'acquire' stage 2019-02-14 22:35:47,147 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2019-02-14 22:35:47,147 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2019-02-14 22:35:47,147 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool7-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:47,148 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool7-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. started... 2019-02-14 22:35:47,148 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool7-thread-1] regionserver.HRegion(2432): Flushing 1/1 column families, memstore=120 B 2019-02-14 22:35:47,167 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool7-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f 2019-02-14 22:35:47,167 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool7-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f 2019-02-14 22:35:47,170 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741838_1014{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 4467 2019-02-14 22:35:47,191 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f3ca40dc-91f2-42ec-82fe-8f7a3a14736f:NORMAL:127.0.0.1:38317|FINALIZED]]} size 0 2019-02-14 22:35:47,192 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool7-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=17, memsize=120, hasBloomFilter=true, into tmp file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f/9c78dc806a1a49d0b290b58c2195b9ee 2019-02-14 22:35:47,200 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool7-thread-1] regionserver.StoreFile(495): HFile Bloom filter type for 9c78dc806a1a49d0b290b58c2195b9ee: NONE, but ROW specified in column family configuration 2019-02-14 22:35:47,201 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool7-thread-1] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f/9c78dc806a1a49d0b290b58c2195b9ee as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/9c78dc806a1a49d0b290b58c2195b9ee 2019-02-14 22:35:47,209 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool7-thread-1] regionserver.StoreFile(495): HFile Bloom filter type for 9c78dc806a1a49d0b290b58c2195b9ee: NONE, but ROW specified in column family configuration 2019-02-14 22:35:47,210 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool7-thread-1] regionserver.HStore(1050): Added hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/9c78dc806a1a49d0b290b58c2195b9ee, entries=0, sequenceid=17, filesize=4.4 K 2019-02-14 22:35:47,211 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool7-thread-1] regionserver.HRegion(2732): Finished memstore flush of ~120 B/120, currentsize=0 B/0 for region testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. in 63ms, sequenceid=17, compaction requested=true 2019-02-14 22:35:47,212 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool7-thread-1] regionserver.HRegion(2233): Flush status journal: Acquiring readlock on region at 1550183747148 Running coprocessor pre-flush hooks at 1550183747148 Obtaining lock to block concurrent updates at 1550183747148 Preparing to flush by snapshotting stores in 39958fd836f676bdcfb70cc61f4ef4c0 at 1550183747148 Finished memstore snapshotting testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0., syncing WAL and waiting on mvcc, flushsize=120 at 1550183747148 Flushing stores of testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. at 1550183747162 Flushing f: creating writer at 1550183747163 Flushing f: appending metadata at 1550183747172 Flushing f: closing flushed file at 1550183747172 Flushing f: reopening flushed file at 1550183747202 Finished memstore flush of ~120 B/120, currentsize=0 B/0 for region testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. in 63ms, sequenceid=17, compaction requested=true at 1550183747211 Running post-flush coprocessor hooks at 1550183747211 Flush successful at 1550183747212 2019-02-14 22:35:47,212 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool7-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:47,212 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2019-02-14 22:35:47,212 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2019-02-14 22:35:47,212 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,212 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' locally acquired 2019-02-14 22:35:47,212 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure (testFIFOCompactionPolicyExpiredEmptyHFiles) in zk 2019-02-14 22:35:47,215 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,215 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,215 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,215 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:47,215 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:47,215 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,215 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2019-02-14 22:35:47,216 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:47,216 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:47,216 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,216 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,217 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:47,217 DEBUG [main-EventThread] procedure.Procedure(298): member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' on coordinator 2019-02-14 22:35:47,217 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@6064c1a4[Count = 0] remaining members to acquire global barrier 2019-02-14 22:35:47,217 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' starting 'in-barrier' execution. 2019-02-14 22:35:47,217 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,219 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,219 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,219 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,219 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' received 'reached' from coordinator. 2019-02-14 22:35:47,219 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' locally completed 2019-02-14 22:35:47,219 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' completed for member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' in zk 2019-02-14 22:35:47,219 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,219 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2019-02-14 22:35:47,222 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,222 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' has notified controller of completion 2019-02-14 22:35:47,222 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:47,222 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' completed. 2019-02-14 22:35:47,222 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,223 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:47,223 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:47,223 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:47,223 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:47,224 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,224 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,224 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:47,225 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,225 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,225 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985': 2019-02-14 22:35:47,226 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' released barrier for procedure'testFIFOCompactionPolicyExpiredEmptyHFiles', counting down latch. Waiting for 0 more 2019-02-14 22:35:47,226 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'testFIFOCompactionPolicyExpiredEmptyHFiles' execution completed 2019-02-14 22:35:47,226 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2019-02-14 22:35:47,226 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2019-02-14 22:35:47,226 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,226 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure testFIFOCompactionPolicyExpiredEmptyHFilesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2019-02-14 22:35:47,228 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,228 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,228 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,228 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,228 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,228 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:47,228 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:47,228 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,228 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:47,228 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:47,228 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:47,229 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:47,229 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,229 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,230 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:47,230 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,230 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,231 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,231 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:47,231 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,233 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:47,233 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:47,233 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:47,233 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:47,233 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'testFIFOCompactionPolicyExpiredEmptyHFiles' 2019-02-14 22:35:47,234 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2019-02-14 22:35:47,234 DEBUG [main-EventThread] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles because node does not exist (not an error) 2019-02-14 22:35:47,234 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:47,235 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:47,235 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:47,235 DEBUG [main] client.HBaseAdmin(4115): Waiting a max of 300000 ms for procedure 'flush-table-proc : testFIFOCompactionPolicyExpiredEmptyHFiles'' to complete. (max 8571 ms per retry) 2019-02-14 22:35:47,235 DEBUG [main] client.HBaseAdmin(4124): (#1) Sleeping: 100ms while waiting for procedure completion. 2019-02-14 22:35:47,234 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,236 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,236 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,236 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:47,236 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,236 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,236 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testFIFOCompactionPolicyExpiredEmptyHFiles 2019-02-14 22:35:47,335 DEBUG [main] client.HBaseAdmin(4130): Getting current status of procedure from master... 2019-02-14 22:35:47,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(989): Checking to see if procedure from request:flush-table-proc is done 2019-02-14 22:35:47,339 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hbase-admin-on-hconnection-0x39e67516 connecting to ZooKeeper ensemble=localhost:50587 2019-02-14 22:35:47,342 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.RSRpcServices(1480): Compacting testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:47,343 ERROR [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] ipc.RpcServer(2437): Unexpected throwable object java.lang.IllegalArgumentException: [hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/2456003b842340a8a586d3721c8cc5e1, hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/9c78dc806a1a49d0b290b58c2195b9ee] overlaps with [hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/2456003b842340a8a586d3721c8cc5e1] at com.google.common.base.Preconditions.checkArgument(Preconditions.java:119) at org.apache.hadoop.hbase.regionserver.HStore.addToCompactingFiles(HStore.java:1824) at org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1798) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:415) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:388) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:317) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:306) at org.apache.hadoop.hbase.regionserver.RSRpcServices.compactRegion(RSRpcServices.java:1513) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:23649) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) 2019-02-14 22:35:47,343 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): hbase-admin-on-hconnection-0x39e675160x0, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2019-02-14 22:35:47,343 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] ipc.CallRunner(130): RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948: callId: 43 service: AdminService methodName: CompactRegion size: 126 connection: 172.31.47.35:51830 deadline: 1550183807342 java.io.IOException: [hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/2456003b842340a8a586d3721c8cc5e1, hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/9c78dc806a1a49d0b290b58c2195b9ee] overlaps with [hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/2456003b842340a8a586d3721c8cc5e1] at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2438) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) Caused by: java.lang.IllegalArgumentException: [hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/2456003b842340a8a586d3721c8cc5e1, hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/9c78dc806a1a49d0b290b58c2195b9ee] overlaps with [hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/2456003b842340a8a586d3721c8cc5e1] at com.google.common.base.Preconditions.checkArgument(Preconditions.java:119) at org.apache.hadoop.hbase.regionserver.HStore.addToCompactingFiles(HStore.java:1824) at org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1798) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:415) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:388) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:317) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:306) at org.apache.hadoop.hbase.regionserver.RSRpcServices.compactRegion(RSRpcServices.java:1513) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:23649) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380) ... 3 more 2019-02-14 22:35:47,343 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(678): hbase-admin-on-hconnection-0x39e67516-0x168ee26a5f9000b connected 2019-02-14 22:35:47,376 INFO [main] hbase.ResourceChecker(171): after: regionserver.compactions.TestFIFOCompactionPolicy#testFIFOCompactionPolicyExpiredEmptyHFiles Thread=249 (was 226) Potentially hanging thread: hconnection-0x5774ef9b-shared--pool8-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: htable-pool17-t1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: AM.ZK.Worker-pool5-t13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: (ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2523) org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2480) org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2445) org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106) org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.finishClose(AbstractHFileWriter.java:248) org.apache.hadoop.hbase.io.hfile.HFileWriterV3.finishClose(HFileWriterV3.java:134) org.apache.hadoop.hbase.io.hfile.HFileWriterV2.close(HFileWriterV2.java:392) org.apache.hadoop.hbase.regionserver.StoreFile$Writer.close(StoreFile.java:1129) org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.commitWriter(DefaultCompactor.java:91) org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.commitWriter(DefaultCompactor.java:43) org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:343) org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:66) org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:115) org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1361) org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2105) org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:579) org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:619) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: AM.ZK.Worker-pool5-t12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x5774ef9b-shared--pool8-t12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x5774ef9b-shared--pool8-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataStreamer for file /user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f/5efb803b5c714e3c8c8024b214596219 java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:523) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x1cb7936c-metaLookup-shared--pool18-t1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_OPEN_REGION-ip-172-31-47-35:39948-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_725136821_1 at /127.0.0.1:37826 [Waiting for operation #17] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x173a5a42-shared--pool11-t2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: AM.ZK.Worker-pool5-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x5774ef9b-shared--pool8-t13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: AM.ZK.Worker-pool5-t9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: AM.ZK.Worker-pool5-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=564 (was 546) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=65536 (was 65536), SystemLoadAverage=270 (was 270), ProcessCount=429 (was 429), AvailableMemoryMB=8683 (was 9156), ConnectionCount=1 (was 0) - ConnectionCount LEAK? - 2019-02-14 22:35:47,395 INFO [main] hbase.ResourceChecker(147): before: regionserver.compactions.TestFIFOCompactionPolicy#testPurgeExpiredFiles Thread=249, OpenFileDescriptor=564, MaxFileDescriptor=65536, SystemLoadAverage=270, ProcessCount=429, AvailableMemoryMB=8682, ConnectionCount=1 2019-02-14 22:35:47,397 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.HMaster$1(1749): Client=apurtell//172.31.47.35 create 'TestFIFOCompactionPolicy', {CONFIGURATION => {'hbase.hstore.defaultengine.compactionpolicy.class' => 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy', 'hbase.regionserver.region.split.policy' => 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy'}}, {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => '1 SECOND', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2019-02-14 22:35:47,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure2.ProcedureExecutor(769): Procedure CreateTableProcedure (table=TestFIFOCompactionPolicy) id=5 owner=apurtell state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2019-02-14 22:35:47,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(1059): Checking to see if procedure is done procId=5 2019-02-14 22:35:47,507 DEBUG [ProcedureExecutor-4] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestFIFOCompactionPolicy/write-master:409250000000000 2019-02-14 22:35:47,510 DEBUG [ProcedureExecutor-4] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/table/TestFIFOCompactionPolicy because node does not exist (not an error) 2019-02-14 22:35:47,574 INFO [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.StoreFile(495): HFile Bloom filter type for 5efb803b5c714e3c8c8024b214596219: NONE, but ROW specified in column family configuration 2019-02-14 22:35:47,575 DEBUG [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/.tmp/f/5efb803b5c714e3c8c8024b214596219 as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/5efb803b5c714e3c8c8024b214596219 2019-02-14 22:35:47,581 INFO [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.StoreFile(495): HFile Bloom filter type for 5efb803b5c714e3c8c8024b214596219: NONE, but ROW specified in column family configuration 2019-02-14 22:35:47,582 DEBUG [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.HStore(1938): Completing compaction... 2019-02-14 22:35:47,583 INFO [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.HStore(1517): Completed compaction of 1 file(s) in f of testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. into 5efb803b5c714e3c8c8024b214596219(size=4.4 K), total size for store is 8.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2019-02-14 22:35:47,583 DEBUG [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.HRegion(2127): Compaction status journal: Compacting store f at 1550183747094 Compaction complete at 1550183747583 2019-02-14 22:35:47,584 INFO [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.CompactSplitThread$CompactionRunner(581): Completed compaction: Request = regionName=testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0., storeName=f, fileCount=1, fileSize=4.4 K (4.4 K), priority=1, time=36216645862130501; duration=0sec 2019-02-14 22:35:47,584 DEBUG [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.CompactSplitThread$CompactionRunner(606): CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2019-02-14 22:35:47,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(1059): Checking to see if procedure is done procId=5 2019-02-14 22:35:47,626 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741840_1016{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 522 2019-02-14 22:35:47,729 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(146): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2019-02-14 22:35:47,729 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(150): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2019-02-14 22:35:47,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(1059): Checking to see if procedure is done procId=5 2019-02-14 22:35:48,029 DEBUG [ProcedureExecutor-4] util.FSTableDescriptors(660): Wrote descriptor into: hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/.tmp/data/default/TestFIFOCompactionPolicy/.tabledesc/.tableinfo.0000000001 2019-02-14 22:35:48,030 INFO [RegionOpenAndInitThread-TestFIFOCompactionPolicy-1] regionserver.HRegion(6831): creating HRegion TestFIFOCompactionPolicy HTD == 'TestFIFOCompactionPolicy', {CONFIGURATION => {'hbase.hstore.defaultengine.compactionpolicy.class' => 'org.apache.hadoop.hbase.regionserver.compactions.FIFOCompactionPolicy', 'hbase.regionserver.region.split.policy' => 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy'}}, {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => '1 SECOND', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/.tmp Table name == TestFIFOCompactionPolicy 2019-02-14 22:35:48,040 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741841_1017{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f3ca40dc-91f2-42ec-82fe-8f7a3a14736f:NORMAL:127.0.0.1:38317|RBW]]} size 59 2019-02-14 22:35:48,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(1059): Checking to see if procedure is done procId=5 2019-02-14 22:35:48,442 DEBUG [RegionOpenAndInitThread-TestFIFOCompactionPolicy-1] regionserver.HRegion(826): Instantiated TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:48,442 DEBUG [RegionOpenAndInitThread-TestFIFOCompactionPolicy-1] regionserver.HRegion(1526): Closing TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.: disabling compactions & flushes 2019-02-14 22:35:48,442 DEBUG [RegionOpenAndInitThread-TestFIFOCompactionPolicy-1] regionserver.HRegion(1553): Updates disabled for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:48,442 INFO [RegionOpenAndInitThread-TestFIFOCompactionPolicy-1] regionserver.HRegion(1660): Closed TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:48,551 INFO [ProcedureExecutor-4] hbase.MetaTableAccessor(1234): Added 1 2019-02-14 22:35:48,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(1059): Checking to see if procedure is done procId=5 2019-02-14 22:35:48,657 INFO [ProcedureExecutor-4] zookeeper.ZKTableStateManager(100): Moving table TestFIFOCompactionPolicy state from null to ENABLING 2019-02-14 22:35:48,661 INFO [ProcedureExecutor-4] master.AssignmentManager(1724): Assigning 1 region(s) to ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:48,661 DEBUG [ProcedureExecutor-4] zookeeper.ZKAssign(174): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Async create of unassigned node 0b19a16b6042d69e3048c196d2a7c525 with OFFLINE state 2019-02-14 22:35:48,663 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2019-02-14 22:35:48,663 DEBUG [main-EventThread] master.OfflineCallback(69): rs={0b19a16b6042d69e3048c196d2a7c525 state=OFFLINE, ts=1550183748661, server=null}, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:48,663 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={0b19a16b6042d69e3048c196d2a7c525 state=OFFLINE, ts=1550183748661, server=null}, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:48,666 DEBUG [ProcedureExecutor-4] master.AssignmentManager(1777): ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 unassigned znodes=1 of total=1; oldCounter=0 2019-02-14 22:35:48,667 INFO [ProcedureExecutor-4] master.RegionStates(1257): Transition {0b19a16b6042d69e3048c196d2a7c525 state=OFFLINE, ts=1550183748661, server=null} to {0b19a16b6042d69e3048c196d2a7c525 state=PENDING_OPEN, ts=1550183748667, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:48,667 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.RSRpcServices(1840): Open TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:48,672 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-1] zookeeper.ZKAssign(805): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioning 0b19a16b6042d69e3048c196d2a7c525 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2019-02-14 22:35:48,673 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/0b19a16b6042d69e3048c196d2a7c525 2019-02-14 22:35:48,673 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-1] zookeeper.ZKAssign(880): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioned node 0b19a16b6042d69e3048c196d2a7c525 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2019-02-14 22:35:48,674 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-1] regionserver.HRegion(7020): Opening region: {ENCODED => 0b19a16b6042d69e3048c196d2a7c525, NAME => 'TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.', STARTKEY => '', ENDKEY => ''} 2019-02-14 22:35:48,674 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table TestFIFOCompactionPolicy 0b19a16b6042d69e3048c196d2a7c525 2019-02-14 22:35:48,674 DEBUG [AM.ZK.Worker-pool5-t15] master.AssignmentManager(1052): Handling RS_ZK_REGION_OPENING, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985, region=0b19a16b6042d69e3048c196d2a7c525, current_state={0b19a16b6042d69e3048c196d2a7c525 state=PENDING_OPEN, ts=1550183748667, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:48,674 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-1] regionserver.HRegion(826): Instantiated TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:48,674 INFO [AM.ZK.Worker-pool5-t15] master.RegionStates(1257): Transition {0b19a16b6042d69e3048c196d2a7c525 state=PENDING_OPEN, ts=1550183748667, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} to {0b19a16b6042d69e3048c196d2a7c525 state=OPENING, ts=1550183748674, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:48,678 DEBUG [StoreOpener-0b19a16b6042d69e3048c196d2a7c525-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f 2019-02-14 22:35:48,678 DEBUG [StoreOpener-0b19a16b6042d69e3048c196d2a7c525-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f 2019-02-14 22:35:48,678 INFO [StoreOpener-0b19a16b6042d69e3048c196d2a7c525-1] hfile.CacheConfig(242): Created cacheConfig for f: blockCache=LruBlockCache{blockCount=0, currentSize=765640, freeSize=1043196664, maxSize=1043962304, heapSize=765640, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2019-02-14 22:35:48,678 INFO [StoreOpener-0b19a16b6042d69e3048c196d2a7c525-1] compactions.CompactionConfiguration(149): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2019-02-14 22:35:48,679 INFO [StoreOpener-0b19a16b6042d69e3048c196d2a7c525-1] regionserver.HStore(370): Store=f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, encoding=NONE, compression=NONE 2019-02-14 22:35:48,680 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-1] regionserver.HRegion(4352): Found 0 recovered edits file(s) under hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525 2019-02-14 22:35:48,681 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-1] regionserver.HRegion(4352): Found 0 recovered edits file(s) under hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525 2019-02-14 22:35:48,683 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2019-02-14 22:35:48,686 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-1] wal.WALSplitter(751): Wrote region seqId=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2019-02-14 22:35:48,686 INFO [RS_OPEN_REGION-ip-172-31-47-35:39948-1] regionserver.HRegion(977): Onlined 0b19a16b6042d69e3048c196d2a7c525; next sequenceid=2 2019-02-14 22:35:48,686 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-1] zookeeper.ZKAssign(643): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Attempting to retransition opening state of node 0b19a16b6042d69e3048c196d2a7c525 2019-02-14 22:35:48,687 INFO [PostOpenDeployTasks:0b19a16b6042d69e3048c196d2a7c525] regionserver.HRegionServer(2058): Post open deploy tasks for TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:48,689 INFO [PostOpenDeployTasks:0b19a16b6042d69e3048c196d2a7c525] hbase.MetaTableAccessor(1434): Updated row TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. with server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:48,689 DEBUG [PostOpenDeployTasks:0b19a16b6042d69e3048c196d2a7c525] regionserver.HRegionServer(2092): Finished post open deploy task for TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:48,689 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-1] zookeeper.ZKAssign(805): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioning 0b19a16b6042d69e3048c196d2a7c525 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2019-02-14 22:35:48,691 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/0b19a16b6042d69e3048c196d2a7c525 2019-02-14 22:35:48,691 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-1] zookeeper.ZKAssign(880): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Transitioned node 0b19a16b6042d69e3048c196d2a7c525 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2019-02-14 22:35:48,691 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-1] coordination.ZkOpenRegionCoordination(91): Transitioned 0b19a16b6042d69e3048c196d2a7c525 to OPENED in zk on ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:48,691 DEBUG [RS_OPEN_REGION-ip-172-31-47-35:39948-1] handler.OpenRegionHandler(172): Opened TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. on ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:48,692 DEBUG [AM.ZK.Worker-pool5-t16] master.AssignmentManager(1052): Handling RS_ZK_REGION_OPENED, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985, region=0b19a16b6042d69e3048c196d2a7c525, current_state={0b19a16b6042d69e3048c196d2a7c525 state=OPENING, ts=1550183748674, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:48,692 INFO [AM.ZK.Worker-pool5-t16] master.RegionStates(1257): Transition {0b19a16b6042d69e3048c196d2a7c525 state=OPENING, ts=1550183748674, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} to {0b19a16b6042d69e3048c196d2a7c525 state=OPEN, ts=1550183748692, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:48,692 DEBUG [AM.ZK.Worker-pool5-t16] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 0b19a16b6042d69e3048c196d2a7c525 from ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583; deleting unassigned node 2019-02-14 22:35:48,694 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/0b19a16b6042d69e3048c196d2a7c525 2019-02-14 22:35:48,694 DEBUG [AM.ZK.Worker-pool5-t16] zookeeper.ZKAssign(479): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Deleted unassigned node 0b19a16b6042d69e3048c196d2a7c525 in expected state RS_ZK_REGION_OPENED 2019-02-14 22:35:48,694 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2019-02-14 22:35:48,694 DEBUG [AM.ZK.Worker-pool5-t17] master.AssignmentManager$5(1448): Znode TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. deleted, state: {0b19a16b6042d69e3048c196d2a7c525 state=OPEN, ts=1550183748692, server=ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985} 2019-02-14 22:35:48,694 DEBUG [AM.ZK.Worker-pool5-t17] master.RegionStates(553): Onlined 0b19a16b6042d69e3048c196d2a7c525 on ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:48,695 DEBUG [ProcedureExecutor-4] master.AssignmentManager(1919): Bulk assigning done for ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:48,695 INFO [ProcedureExecutor-4] zookeeper.ZKTableStateManager(100): Moving table TestFIFOCompactionPolicy state from ENABLING to ENABLED 2019-02-14 22:35:49,010 DEBUG [ProcedureExecutor-4] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestFIFOCompactionPolicy/write-master:409250000000000 2019-02-14 22:35:49,010 DEBUG [ProcedureExecutor-4] procedure2.ProcedureExecutor(970): Procedure completed in 1.5070sec: CreateTableProcedure (table=TestFIFOCompactionPolicy) id=5 owner=apurtell state=FINISHED 2019-02-14 22:35:49,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(1059): Checking to see if procedure is done procId=5 2019-02-14 22:35:49,616 INFO [main] client.HBaseAdmin$CreateTableFuture(795): Created TestFIFOCompactionPolicy 2019-02-14 22:35:49,659 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(753): Client=apurtell//172.31.47.35 procedure request for: flush-table-proc 2019-02-14 22:35:49,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(177): Submitting procedure TestFIFOCompactionPolicy 2019-02-14 22:35:49,661 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'TestFIFOCompactionPolicy' 2019-02-14 22:35:49,661 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:49,662 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'TestFIFOCompactionPolicy' starting 'acquire' 2019-02-14 22:35:49,662 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'TestFIFOCompactionPolicy', kicking off acquire phase on members. 2019-02-14 22:35:49,662 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:49,662 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:49,664 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:49,664 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,664 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:49,664 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:49,664 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,664 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2019-02-14 22:35:49,664 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:49,664 DEBUG [main-EventThread] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:49,665 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2019-02-14 22:35:49,665 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:49,665 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestFIFOCompactionPolicy 2019-02-14 22:35:49,665 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:TestFIFOCompactionPolicy 2019-02-14 22:35:49,665 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'TestFIFOCompactionPolicy' with timeout 60000ms 2019-02-14 22:35:49,665 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:49,666 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'TestFIFOCompactionPolicy' starting 'acquire' stage 2019-02-14 22:35:49,666 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2019-02-14 22:35:49,666 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2019-02-14 22:35:49,666 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool8-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:49,666 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool8-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. started... 2019-02-14 22:35:49,667 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool8-thread-1] regionserver.HRegion(2432): Flushing 1/1 column families, memstore=1.25 MB 2019-02-14 22:35:49,670 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool8-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:49,670 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool8-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:49,687 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741842_1018{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 0 2019-02-14 22:35:49,688 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool8-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=15, memsize=1.3 M, hasBloomFilter=true, into tmp file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/9d1822e1d3a447c3a6fdc07c8d3a3c7b 2019-02-14 22:35:49,697 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool8-thread-1] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/9d1822e1d3a447c3a6fdc07c8d3a3c7b as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/9d1822e1d3a447c3a6fdc07c8d3a3c7b 2019-02-14 22:35:49,703 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool8-thread-1] regionserver.HStore(1050): Added hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/9d1822e1d3a447c3a6fdc07c8d3a3c7b, entries=10, sequenceid=15, filesize=1.3 M 2019-02-14 22:35:49,705 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool8-thread-1] regionserver.HRegion(2732): Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 39ms, sequenceid=15, compaction requested=false 2019-02-14 22:35:49,705 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool8-thread-1] regionserver.HRegion(2233): Flush status journal: Acquiring readlock on region at 1550183749666 Running coprocessor pre-flush hooks at 1550183749666 Obtaining lock to block concurrent updates at 1550183749667 Preparing to flush by snapshotting stores in 0b19a16b6042d69e3048c196d2a7c525 at 1550183749667 Finished memstore snapshotting TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525., syncing WAL and waiting on mvcc, flushsize=1311776 at 1550183749667 Flushing stores of TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. at 1550183749667 Flushing f: creating writer at 1550183749667 Flushing f: appending metadata at 1550183749679 Flushing f: closing flushed file at 1550183749679 Flushing f: reopening flushed file at 1550183749698 Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 39ms, sequenceid=15, compaction requested=false at 1550183749705 Running post-flush coprocessor hooks at 1550183749705 Flush successful at 1550183749705 2019-02-14 22:35:49,705 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool8-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:49,705 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2019-02-14 22:35:49,706 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2019-02-14 22:35:49,706 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,706 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'TestFIFOCompactionPolicy' locally acquired 2019-02-14 22:35:49,707 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure (TestFIFOCompactionPolicy) in zk 2019-02-14 22:35:49,709 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,709 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:49,709 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,709 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:49,709 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:49,710 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:49,710 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'TestFIFOCompactionPolicy' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2019-02-14 22:35:49,710 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:49,710 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:49,710 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:49,710 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,711 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:49,711 DEBUG [main-EventThread] procedure.Procedure(298): member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure 'TestFIFOCompactionPolicy' on coordinator 2019-02-14 22:35:49,711 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@5f536b5c[Count = 0] remaining members to acquire global barrier 2019-02-14 22:35:49,711 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'TestFIFOCompactionPolicy' starting 'in-barrier' execution. 2019-02-14 22:35:49,711 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:49,712 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:49,712 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:49,713 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:49,713 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'TestFIFOCompactionPolicy' received 'reached' from coordinator. 2019-02-14 22:35:49,713 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'TestFIFOCompactionPolicy' locally completed 2019-02-14 22:35:49,713 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'TestFIFOCompactionPolicy' completed for member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' in zk 2019-02-14 22:35:49,713 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,713 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2019-02-14 22:35:49,714 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,714 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,714 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:49,714 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:49,714 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'TestFIFOCompactionPolicy' has notified controller of completion 2019-02-14 22:35:49,715 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:49,715 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'TestFIFOCompactionPolicy' completed. 2019-02-14 22:35:49,715 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:49,715 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:49,716 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:49,717 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,718 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:49,718 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:49,718 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,718 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'TestFIFOCompactionPolicy' member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985': 2019-02-14 22:35:49,719 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' released barrier for procedure'TestFIFOCompactionPolicy', counting down latch. Waiting for 0 more 2019-02-14 22:35:49,719 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'TestFIFOCompactionPolicy' execution completed 2019-02-14 22:35:49,719 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2019-02-14 22:35:49,719 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2019-02-14 22:35:49,719 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:TestFIFOCompactionPolicy 2019-02-14 22:35:49,719 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure TestFIFOCompactionPolicyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2019-02-14 22:35:49,721 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:49,721 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:49,721 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:49,721 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:49,721 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:49,721 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:49,721 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:49,721 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:49,721 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,721 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:49,721 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:49,721 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:49,721 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:49,722 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:49,722 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:49,722 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:49,722 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,722 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,722 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:49,723 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:49,723 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,725 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:49,725 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:49,725 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:49,725 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:49,725 DEBUG [main-EventThread] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy because node does not exist (not an error) 2019-02-14 22:35:49,725 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'TestFIFOCompactionPolicy' 2019-02-14 22:35:49,726 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2019-02-14 22:35:49,726 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:49,726 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:49,726 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:49,725 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,726 DEBUG [main] client.HBaseAdmin(4115): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestFIFOCompactionPolicy'' to complete. (max 8571 ms per retry) 2019-02-14 22:35:49,727 DEBUG [main] client.HBaseAdmin(4124): (#1) Sleeping: 100ms while waiting for procedure completion. 2019-02-14 22:35:49,726 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:49,727 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:49,727 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,727 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:49,727 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:49,727 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:49,827 DEBUG [main] client.HBaseAdmin(4130): Getting current status of procedure from master... 2019-02-14 22:35:49,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(989): Checking to see if procedure from request:flush-table-proc is done 2019-02-14 22:35:49,859 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(753): Client=apurtell//172.31.47.35 procedure request for: flush-table-proc 2019-02-14 22:35:49,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(155): Procedure TestFIFOCompactionPolicy was in running list but was completed. Accepting new attempt. 2019-02-14 22:35:49,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(177): Submitting procedure TestFIFOCompactionPolicy 2019-02-14 22:35:49,863 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'TestFIFOCompactionPolicy' 2019-02-14 22:35:49,863 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:49,863 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'TestFIFOCompactionPolicy' starting 'acquire' 2019-02-14 22:35:49,863 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'TestFIFOCompactionPolicy', kicking off acquire phase on members. 2019-02-14 22:35:49,864 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:49,864 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:49,865 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:49,865 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,865 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:49,865 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:49,866 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:49,866 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2019-02-14 22:35:49,866 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:49,866 DEBUG [main-EventThread] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:49,866 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2019-02-14 22:35:49,866 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:49,866 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestFIFOCompactionPolicy 2019-02-14 22:35:49,866 WARN [main-EventThread] procedure.ProcedureMember(142): A completed old subproc TestFIFOCompactionPolicy is still present, removing 2019-02-14 22:35:49,867 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:TestFIFOCompactionPolicy 2019-02-14 22:35:49,867 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'TestFIFOCompactionPolicy' with timeout 60000ms 2019-02-14 22:35:49,867 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:49,867 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'TestFIFOCompactionPolicy' starting 'acquire' stage 2019-02-14 22:35:49,868 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2019-02-14 22:35:49,868 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2019-02-14 22:35:49,868 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool9-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:49,868 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool9-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. started... 2019-02-14 22:35:49,868 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool9-thread-1] regionserver.HRegion(2432): Flushing 1/1 column families, memstore=1.25 MB 2019-02-14 22:35:49,871 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool9-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:49,871 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool9-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:49,889 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741843_1019{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 1316785 2019-02-14 22:35:50,291 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool9-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=29, memsize=1.3 M, hasBloomFilter=true, into tmp file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/51309c87436c439aa1a005af85f79252 2019-02-14 22:35:50,300 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool9-thread-1] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/51309c87436c439aa1a005af85f79252 as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/51309c87436c439aa1a005af85f79252 2019-02-14 22:35:50,309 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool9-thread-1] regionserver.HStore(1050): Added hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/51309c87436c439aa1a005af85f79252, entries=10, sequenceid=29, filesize=1.3 M 2019-02-14 22:35:50,310 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool9-thread-1] regionserver.HRegion(2732): Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 442ms, sequenceid=29, compaction requested=true 2019-02-14 22:35:50,310 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool9-thread-1] regionserver.HRegion(2233): Flush status journal: Acquiring readlock on region at 1550183749868 Running coprocessor pre-flush hooks at 1550183749868 Obtaining lock to block concurrent updates at 1550183749868 Preparing to flush by snapshotting stores in 0b19a16b6042d69e3048c196d2a7c525 at 1550183749868 Finished memstore snapshotting TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525., syncing WAL and waiting on mvcc, flushsize=1311776 at 1550183749869 Flushing stores of TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. at 1550183749869 Flushing f: creating writer at 1550183749869 Flushing f: appending metadata at 1550183749884 Flushing f: closing flushed file at 1550183749884 Flushing f: reopening flushed file at 1550183750301 Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 442ms, sequenceid=29, compaction requested=true at 1550183750310 Running post-flush coprocessor hooks at 1550183750310 Flush successful at 1550183750310 2019-02-14 22:35:50,310 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool9-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:50,310 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2019-02-14 22:35:50,311 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2019-02-14 22:35:50,311 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,311 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'TestFIFOCompactionPolicy' locally acquired 2019-02-14 22:35:50,311 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure (TestFIFOCompactionPolicy) in zk 2019-02-14 22:35:50,315 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,315 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,315 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,315 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:50,315 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:50,316 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,316 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'TestFIFOCompactionPolicy' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2019-02-14 22:35:50,316 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:50,316 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:50,316 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:50,317 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,317 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:50,317 DEBUG [main-EventThread] procedure.Procedure(298): member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure 'TestFIFOCompactionPolicy' on coordinator 2019-02-14 22:35:50,317 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@1623d6c[Count = 0] remaining members to acquire global barrier 2019-02-14 22:35:50,317 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'TestFIFOCompactionPolicy' starting 'in-barrier' execution. 2019-02-14 22:35:50,317 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,319 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,319 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,319 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,319 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'TestFIFOCompactionPolicy' received 'reached' from coordinator. 2019-02-14 22:35:50,319 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'TestFIFOCompactionPolicy' locally completed 2019-02-14 22:35:50,319 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'TestFIFOCompactionPolicy' completed for member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' in zk 2019-02-14 22:35:50,319 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,320 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2019-02-14 22:35:50,322 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,322 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,322 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:50,322 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:50,322 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'TestFIFOCompactionPolicy' has notified controller of completion 2019-02-14 22:35:50,322 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:50,322 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'TestFIFOCompactionPolicy' completed. 2019-02-14 22:35:50,323 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:50,323 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:50,324 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:50,324 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,324 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:50,324 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:50,325 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,325 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'TestFIFOCompactionPolicy' member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985': 2019-02-14 22:35:50,325 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' released barrier for procedure'TestFIFOCompactionPolicy', counting down latch. Waiting for 0 more 2019-02-14 22:35:50,325 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'TestFIFOCompactionPolicy' execution completed 2019-02-14 22:35:50,326 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2019-02-14 22:35:50,326 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2019-02-14 22:35:50,326 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:TestFIFOCompactionPolicy 2019-02-14 22:35:50,326 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure TestFIFOCompactionPolicyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2019-02-14 22:35:50,328 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:50,328 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:50,329 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:50,329 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:50,329 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:50,329 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:50,329 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:50,329 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,329 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:50,329 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:50,329 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:50,329 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:50,329 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:50,330 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:50,330 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:50,330 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,330 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:50,330 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,331 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:50,331 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:50,333 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:50,333 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:50,333 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:50,333 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:50,333 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:50,333 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:50,333 DEBUG [main-EventThread] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy because node does not exist (not an error) 2019-02-14 22:35:50,333 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'TestFIFOCompactionPolicy' 2019-02-14 22:35:50,333 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2019-02-14 22:35:50,333 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:50,333 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,335 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:50,335 DEBUG [main] client.HBaseAdmin(4115): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestFIFOCompactionPolicy'' to complete. (max 8571 ms per retry) 2019-02-14 22:35:50,335 DEBUG [main] client.HBaseAdmin(4124): (#1) Sleeping: 100ms while waiting for procedure completion. 2019-02-14 22:35:50,335 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:50,335 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,335 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,335 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,335 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:50,435 DEBUG [main] client.HBaseAdmin(4130): Getting current status of procedure from master... 2019-02-14 22:35:50,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(989): Checking to see if procedure from request:flush-table-proc is done 2019-02-14 22:35:50,463 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(753): Client=apurtell//172.31.47.35 procedure request for: flush-table-proc 2019-02-14 22:35:50,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(155): Procedure TestFIFOCompactionPolicy was in running list but was completed. Accepting new attempt. 2019-02-14 22:35:50,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(177): Submitting procedure TestFIFOCompactionPolicy 2019-02-14 22:35:50,466 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'TestFIFOCompactionPolicy' 2019-02-14 22:35:50,466 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:50,467 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'TestFIFOCompactionPolicy' starting 'acquire' 2019-02-14 22:35:50,467 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'TestFIFOCompactionPolicy', kicking off acquire phase on members. 2019-02-14 22:35:50,467 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:50,467 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:50,469 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:50,469 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,469 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:50,469 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:50,469 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,469 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2019-02-14 22:35:50,470 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:50,470 DEBUG [main-EventThread] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:50,470 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2019-02-14 22:35:50,470 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:50,470 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestFIFOCompactionPolicy 2019-02-14 22:35:50,470 WARN [main-EventThread] procedure.ProcedureMember(142): A completed old subproc TestFIFOCompactionPolicy is still present, removing 2019-02-14 22:35:50,471 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:TestFIFOCompactionPolicy 2019-02-14 22:35:50,471 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'TestFIFOCompactionPolicy' with timeout 60000ms 2019-02-14 22:35:50,471 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:50,471 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'TestFIFOCompactionPolicy' starting 'acquire' stage 2019-02-14 22:35:50,471 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2019-02-14 22:35:50,472 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2019-02-14 22:35:50,472 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool10-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:50,472 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool10-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. started... 2019-02-14 22:35:50,472 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool10-thread-1] regionserver.HRegion(2432): Flushing 1/1 column families, memstore=1.25 MB 2019-02-14 22:35:50,474 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool10-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:50,474 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool10-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:50,488 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741844_1020{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f3ca40dc-91f2-42ec-82fe-8f7a3a14736f:NORMAL:127.0.0.1:38317|RBW]]} size 1316785 2019-02-14 22:35:50,889 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool10-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=43, memsize=1.3 M, hasBloomFilter=true, into tmp file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/6c664385246141a4bb1aa0301536b8f5 2019-02-14 22:35:50,898 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool10-thread-1] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/6c664385246141a4bb1aa0301536b8f5 as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/6c664385246141a4bb1aa0301536b8f5 2019-02-14 22:35:50,905 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool10-thread-1] regionserver.HStore(1050): Added hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/6c664385246141a4bb1aa0301536b8f5, entries=10, sequenceid=43, filesize=1.3 M 2019-02-14 22:35:50,906 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool10-thread-1] regionserver.HRegion(2732): Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 434ms, sequenceid=43, compaction requested=true 2019-02-14 22:35:50,907 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool10-thread-1] regionserver.HRegion(2233): Flush status journal: Acquiring readlock on region at 1550183750472 Running coprocessor pre-flush hooks at 1550183750472 Obtaining lock to block concurrent updates at 1550183750472 Preparing to flush by snapshotting stores in 0b19a16b6042d69e3048c196d2a7c525 at 1550183750472 Finished memstore snapshotting TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525., syncing WAL and waiting on mvcc, flushsize=1311776 at 1550183750472 Flushing stores of TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. at 1550183750473 Flushing f: creating writer at 1550183750473 Flushing f: appending metadata at 1550183750483 Flushing f: closing flushed file at 1550183750483 Flushing f: reopening flushed file at 1550183750899 Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 434ms, sequenceid=43, compaction requested=true at 1550183750907 Running post-flush coprocessor hooks at 1550183750907 Flush successful at 1550183750907 2019-02-14 22:35:50,907 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool10-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:50,907 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2019-02-14 22:35:50,908 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2019-02-14 22:35:50,908 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,908 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'TestFIFOCompactionPolicy' locally acquired 2019-02-14 22:35:50,908 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure (TestFIFOCompactionPolicy) in zk 2019-02-14 22:35:50,912 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,912 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,912 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,912 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:50,912 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:50,912 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,912 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'TestFIFOCompactionPolicy' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2019-02-14 22:35:50,912 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:50,912 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:50,913 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:50,913 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,913 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:50,913 DEBUG [main-EventThread] procedure.Procedure(298): member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure 'TestFIFOCompactionPolicy' on coordinator 2019-02-14 22:35:50,913 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@6599db2b[Count = 0] remaining members to acquire global barrier 2019-02-14 22:35:50,913 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'TestFIFOCompactionPolicy' starting 'in-barrier' execution. 2019-02-14 22:35:50,914 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,915 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,915 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,915 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,915 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'TestFIFOCompactionPolicy' received 'reached' from coordinator. 2019-02-14 22:35:50,915 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'TestFIFOCompactionPolicy' locally completed 2019-02-14 22:35:50,915 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'TestFIFOCompactionPolicy' completed for member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' in zk 2019-02-14 22:35:50,915 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,915 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2019-02-14 22:35:50,917 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,917 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,917 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:50,917 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:50,917 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'TestFIFOCompactionPolicy' has notified controller of completion 2019-02-14 22:35:50,917 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:50,917 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'TestFIFOCompactionPolicy' completed. 2019-02-14 22:35:50,918 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:50,918 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:50,918 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:50,919 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,919 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:50,919 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:50,919 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,920 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'TestFIFOCompactionPolicy' member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985': 2019-02-14 22:35:50,920 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' released barrier for procedure'TestFIFOCompactionPolicy', counting down latch. Waiting for 0 more 2019-02-14 22:35:50,920 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'TestFIFOCompactionPolicy' execution completed 2019-02-14 22:35:50,920 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2019-02-14 22:35:50,920 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2019-02-14 22:35:50,920 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:TestFIFOCompactionPolicy 2019-02-14 22:35:50,920 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure TestFIFOCompactionPolicyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2019-02-14 22:35:50,922 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:50,922 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:50,922 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:50,922 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:50,922 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:50,922 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:50,922 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:50,922 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:50,922 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:50,922 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:50,922 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:50,923 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:50,923 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,923 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:50,923 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:50,923 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:50,923 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,924 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:50,924 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,924 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:50,924 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,926 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,926 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:50,926 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:50,926 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:50,926 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'TestFIFOCompactionPolicy' 2019-02-14 22:35:50,926 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2019-02-14 22:35:50,926 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:50,926 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:50,926 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:50,927 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:50,926 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:50,927 DEBUG [main] client.HBaseAdmin(4115): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestFIFOCompactionPolicy'' to complete. (max 8571 ms per retry) 2019-02-14 22:35:50,927 DEBUG [main] client.HBaseAdmin(4124): (#1) Sleeping: 100ms while waiting for procedure completion. 2019-02-14 22:35:50,926 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:50,927 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:50,928 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,928 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:50,928 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:51,028 DEBUG [main] client.HBaseAdmin(4130): Getting current status of procedure from master... 2019-02-14 22:35:51,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(989): Checking to see if procedure from request:flush-table-proc is done 2019-02-14 22:35:51,058 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(753): Client=apurtell//172.31.47.35 procedure request for: flush-table-proc 2019-02-14 22:35:51,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(155): Procedure TestFIFOCompactionPolicy was in running list but was completed. Accepting new attempt. 2019-02-14 22:35:51,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(177): Submitting procedure TestFIFOCompactionPolicy 2019-02-14 22:35:51,061 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'TestFIFOCompactionPolicy' 2019-02-14 22:35:51,061 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:51,062 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'TestFIFOCompactionPolicy' starting 'acquire' 2019-02-14 22:35:51,062 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'TestFIFOCompactionPolicy', kicking off acquire phase on members. 2019-02-14 22:35:51,062 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:51,062 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:51,064 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:51,064 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,064 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:51,064 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:51,064 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,064 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2019-02-14 22:35:51,064 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:51,064 DEBUG [main-EventThread] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:51,065 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2019-02-14 22:35:51,065 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:51,065 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestFIFOCompactionPolicy 2019-02-14 22:35:51,065 WARN [main-EventThread] procedure.ProcedureMember(142): A completed old subproc TestFIFOCompactionPolicy is still present, removing 2019-02-14 22:35:51,065 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:TestFIFOCompactionPolicy 2019-02-14 22:35:51,065 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'TestFIFOCompactionPolicy' with timeout 60000ms 2019-02-14 22:35:51,065 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:51,066 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'TestFIFOCompactionPolicy' starting 'acquire' stage 2019-02-14 22:35:51,066 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2019-02-14 22:35:51,066 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2019-02-14 22:35:51,066 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool11-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:51,066 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool11-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. started... 2019-02-14 22:35:51,066 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool11-thread-1] regionserver.HRegion(2432): Flushing 1/1 column families, memstore=1.25 MB 2019-02-14 22:35:51,068 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool11-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:51,069 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool11-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:51,086 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741845_1021{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 1316785 2019-02-14 22:35:51,487 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool11-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=57, memsize=1.3 M, hasBloomFilter=true, into tmp file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/ae7b89c41de44052a04c52f21a98c14c 2019-02-14 22:35:51,495 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool11-thread-1] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/ae7b89c41de44052a04c52f21a98c14c as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/ae7b89c41de44052a04c52f21a98c14c 2019-02-14 22:35:51,503 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool11-thread-1] regionserver.HStore(1050): Added hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/ae7b89c41de44052a04c52f21a98c14c, entries=10, sequenceid=57, filesize=1.3 M 2019-02-14 22:35:51,504 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool11-thread-1] regionserver.HRegion(2732): Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 438ms, sequenceid=57, compaction requested=true 2019-02-14 22:35:51,504 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool11-thread-1] regionserver.HRegion(2233): Flush status journal: Acquiring readlock on region at 1550183751066 Running coprocessor pre-flush hooks at 1550183751066 Obtaining lock to block concurrent updates at 1550183751066 Preparing to flush by snapshotting stores in 0b19a16b6042d69e3048c196d2a7c525 at 1550183751066 Finished memstore snapshotting TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525., syncing WAL and waiting on mvcc, flushsize=1311776 at 1550183751066 Flushing stores of TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. at 1550183751067 Flushing f: creating writer at 1550183751067 Flushing f: appending metadata at 1550183751081 Flushing f: closing flushed file at 1550183751081 Flushing f: reopening flushed file at 1550183751496 Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 438ms, sequenceid=57, compaction requested=true at 1550183751504 Running post-flush coprocessor hooks at 1550183751504 Flush successful at 1550183751504 2019-02-14 22:35:51,504 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool11-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:51,504 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2019-02-14 22:35:51,505 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2019-02-14 22:35:51,505 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,506 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'TestFIFOCompactionPolicy' locally acquired 2019-02-14 22:35:51,506 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure (TestFIFOCompactionPolicy) in zk 2019-02-14 22:35:51,508 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,508 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:51,508 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,508 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:51,508 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:51,508 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:51,508 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'TestFIFOCompactionPolicy' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2019-02-14 22:35:51,508 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:51,509 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:51,509 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:51,509 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,509 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:51,510 DEBUG [main-EventThread] procedure.Procedure(298): member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure 'TestFIFOCompactionPolicy' on coordinator 2019-02-14 22:35:51,510 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@4678ab3[Count = 0] remaining members to acquire global barrier 2019-02-14 22:35:51,510 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'TestFIFOCompactionPolicy' starting 'in-barrier' execution. 2019-02-14 22:35:51,510 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:51,511 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:51,511 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:51,511 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:51,511 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'TestFIFOCompactionPolicy' received 'reached' from coordinator. 2019-02-14 22:35:51,512 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'TestFIFOCompactionPolicy' locally completed 2019-02-14 22:35:51,512 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'TestFIFOCompactionPolicy' completed for member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' in zk 2019-02-14 22:35:51,512 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,512 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2019-02-14 22:35:51,513 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,513 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'TestFIFOCompactionPolicy' has notified controller of completion 2019-02-14 22:35:51,513 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:51,513 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,513 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:51,514 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:51,513 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'TestFIFOCompactionPolicy' completed. 2019-02-14 22:35:51,515 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:51,515 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:51,515 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:51,515 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,516 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:51,516 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:51,516 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,517 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'TestFIFOCompactionPolicy' member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985': 2019-02-14 22:35:51,517 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' released barrier for procedure'TestFIFOCompactionPolicy', counting down latch. Waiting for 0 more 2019-02-14 22:35:51,517 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'TestFIFOCompactionPolicy' execution completed 2019-02-14 22:35:51,517 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2019-02-14 22:35:51,517 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2019-02-14 22:35:51,517 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:TestFIFOCompactionPolicy 2019-02-14 22:35:51,517 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure TestFIFOCompactionPolicyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2019-02-14 22:35:51,519 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:51,519 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:51,519 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:51,519 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:51,519 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:51,519 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:51,519 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:51,520 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:51,520 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:51,520 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:51,520 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:51,520 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,520 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:51,520 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:51,520 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:51,521 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:51,521 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,521 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,521 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:51,522 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:51,522 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,523 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:51,523 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:51,523 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:51,523 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'TestFIFOCompactionPolicy' 2019-02-14 22:35:51,523 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2019-02-14 22:35:51,523 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:51,524 DEBUG [main-EventThread] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy because node does not exist (not an error) 2019-02-14 22:35:51,525 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:51,525 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:51,525 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:51,524 DEBUG [main] client.HBaseAdmin(4115): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestFIFOCompactionPolicy'' to complete. (max 8571 ms per retry) 2019-02-14 22:35:51,525 DEBUG [main] client.HBaseAdmin(4124): (#1) Sleeping: 100ms while waiting for procedure completion. 2019-02-14 22:35:51,525 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,525 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:51,525 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:51,525 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,525 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:51,525 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:51,525 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:51,625 DEBUG [main] client.HBaseAdmin(4130): Getting current status of procedure from master... 2019-02-14 22:35:51,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(989): Checking to see if procedure from request:flush-table-proc is done 2019-02-14 22:35:51,661 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(753): Client=apurtell//172.31.47.35 procedure request for: flush-table-proc 2019-02-14 22:35:51,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(155): Procedure TestFIFOCompactionPolicy was in running list but was completed. Accepting new attempt. 2019-02-14 22:35:51,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(177): Submitting procedure TestFIFOCompactionPolicy 2019-02-14 22:35:51,663 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'TestFIFOCompactionPolicy' 2019-02-14 22:35:51,663 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:51,664 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'TestFIFOCompactionPolicy' starting 'acquire' 2019-02-14 22:35:51,664 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'TestFIFOCompactionPolicy', kicking off acquire phase on members. 2019-02-14 22:35:51,665 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:51,665 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:51,667 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:51,667 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,667 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:51,667 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:51,667 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:51,667 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:51,667 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2019-02-14 22:35:51,667 DEBUG [main-EventThread] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:51,668 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2019-02-14 22:35:51,668 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:51,668 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestFIFOCompactionPolicy 2019-02-14 22:35:51,668 WARN [main-EventThread] procedure.ProcedureMember(142): A completed old subproc TestFIFOCompactionPolicy is still present, removing 2019-02-14 22:35:51,668 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:TestFIFOCompactionPolicy 2019-02-14 22:35:51,668 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'TestFIFOCompactionPolicy' with timeout 60000ms 2019-02-14 22:35:51,668 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:51,669 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'TestFIFOCompactionPolicy' starting 'acquire' stage 2019-02-14 22:35:51,669 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2019-02-14 22:35:51,670 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2019-02-14 22:35:51,670 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool12-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:51,670 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool12-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. started... 2019-02-14 22:35:51,670 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool12-thread-1] regionserver.HRegion(2432): Flushing 1/1 column families, memstore=1.25 MB 2019-02-14 22:35:51,673 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool12-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:51,673 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool12-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:51,688 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741846_1022{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 1316785 2019-02-14 22:35:51,887 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2019-02-14 22:35:52,089 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool12-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=71, memsize=1.3 M, hasBloomFilter=true, into tmp file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/34bb22bdc7b247b9b10b363517de408e 2019-02-14 22:35:52,096 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool12-thread-1] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/34bb22bdc7b247b9b10b363517de408e as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/34bb22bdc7b247b9b10b363517de408e 2019-02-14 22:35:52,103 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool12-thread-1] regionserver.HStore(1050): Added hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/34bb22bdc7b247b9b10b363517de408e, entries=10, sequenceid=71, filesize=1.3 M 2019-02-14 22:35:52,104 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool12-thread-1] regionserver.HRegion(2732): Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 434ms, sequenceid=71, compaction requested=true 2019-02-14 22:35:52,104 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool12-thread-1] regionserver.HRegion(2233): Flush status journal: Acquiring readlock on region at 1550183751670 Running coprocessor pre-flush hooks at 1550183751670 Obtaining lock to block concurrent updates at 1550183751670 Preparing to flush by snapshotting stores in 0b19a16b6042d69e3048c196d2a7c525 at 1550183751670 Finished memstore snapshotting TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525., syncing WAL and waiting on mvcc, flushsize=1311776 at 1550183751670 Flushing stores of TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. at 1550183751671 Flushing f: creating writer at 1550183751671 Flushing f: appending metadata at 1550183751683 Flushing f: closing flushed file at 1550183751683 Flushing f: reopening flushed file at 1550183752097 Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 434ms, sequenceid=71, compaction requested=true at 1550183752104 Running post-flush coprocessor hooks at 1550183752104 Flush successful at 1550183752104 2019-02-14 22:35:52,104 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool12-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:52,104 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2019-02-14 22:35:52,105 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2019-02-14 22:35:52,105 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,105 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'TestFIFOCompactionPolicy' locally acquired 2019-02-14 22:35:52,105 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure (TestFIFOCompactionPolicy) in zk 2019-02-14 22:35:52,107 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,107 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,107 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,107 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:52,108 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:52,108 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,108 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'TestFIFOCompactionPolicy' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2019-02-14 22:35:52,108 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:52,108 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:52,108 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:52,109 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,109 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:52,109 DEBUG [main-EventThread] procedure.Procedure(298): member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure 'TestFIFOCompactionPolicy' on coordinator 2019-02-14 22:35:52,109 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@40aa18e8[Count = 0] remaining members to acquire global barrier 2019-02-14 22:35:52,109 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'TestFIFOCompactionPolicy' starting 'in-barrier' execution. 2019-02-14 22:35:52,109 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,111 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,111 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,111 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,111 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'TestFIFOCompactionPolicy' received 'reached' from coordinator. 2019-02-14 22:35:52,111 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'TestFIFOCompactionPolicy' locally completed 2019-02-14 22:35:52,111 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'TestFIFOCompactionPolicy' completed for member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' in zk 2019-02-14 22:35:52,111 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,111 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2019-02-14 22:35:52,114 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,114 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'TestFIFOCompactionPolicy' has notified controller of completion 2019-02-14 22:35:52,114 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:52,114 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,114 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:52,115 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:52,114 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'TestFIFOCompactionPolicy' completed. 2019-02-14 22:35:52,115 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:52,115 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:52,115 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:52,116 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,116 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:52,116 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:52,116 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,117 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'TestFIFOCompactionPolicy' member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985': 2019-02-14 22:35:52,117 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' released barrier for procedure'TestFIFOCompactionPolicy', counting down latch. Waiting for 0 more 2019-02-14 22:35:52,117 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'TestFIFOCompactionPolicy' execution completed 2019-02-14 22:35:52,117 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2019-02-14 22:35:52,117 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2019-02-14 22:35:52,117 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:TestFIFOCompactionPolicy 2019-02-14 22:35:52,117 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure TestFIFOCompactionPolicyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2019-02-14 22:35:52,119 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,119 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,119 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,119 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,119 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,119 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:52,119 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:52,119 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:52,119 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:52,119 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:52,120 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,120 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,120 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:52,120 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:52,120 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:52,120 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,120 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:52,121 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,121 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:52,121 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:52,122 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,123 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:52,123 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:52,123 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:52,123 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:52,123 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'TestFIFOCompactionPolicy' 2019-02-14 22:35:52,124 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2019-02-14 22:35:52,124 DEBUG [main-EventThread] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy because node does not exist (not an error) 2019-02-14 22:35:52,124 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:52,124 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:52,124 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:52,124 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,124 DEBUG [main] client.HBaseAdmin(4115): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestFIFOCompactionPolicy'' to complete. (max 8571 ms per retry) 2019-02-14 22:35:52,124 DEBUG [main] client.HBaseAdmin(4124): (#1) Sleeping: 100ms while waiting for procedure completion. 2019-02-14 22:35:52,124 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:52,124 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:52,124 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,125 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,125 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,125 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,225 DEBUG [main] client.HBaseAdmin(4130): Getting current status of procedure from master... 2019-02-14 22:35:52,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(989): Checking to see if procedure from request:flush-table-proc is done 2019-02-14 22:35:52,256 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(753): Client=apurtell//172.31.47.35 procedure request for: flush-table-proc 2019-02-14 22:35:52,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(155): Procedure TestFIFOCompactionPolicy was in running list but was completed. Accepting new attempt. 2019-02-14 22:35:52,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(177): Submitting procedure TestFIFOCompactionPolicy 2019-02-14 22:35:52,261 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'TestFIFOCompactionPolicy' 2019-02-14 22:35:52,261 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:52,262 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'TestFIFOCompactionPolicy' starting 'acquire' 2019-02-14 22:35:52,262 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'TestFIFOCompactionPolicy', kicking off acquire phase on members. 2019-02-14 22:35:52,262 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,262 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:52,264 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:52,264 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,264 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:52,264 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:52,264 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,265 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2019-02-14 22:35:52,265 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:52,265 DEBUG [main-EventThread] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,265 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2019-02-14 22:35:52,265 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:52,265 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestFIFOCompactionPolicy 2019-02-14 22:35:52,266 WARN [main-EventThread] procedure.ProcedureMember(142): A completed old subproc TestFIFOCompactionPolicy is still present, removing 2019-02-14 22:35:52,266 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:TestFIFOCompactionPolicy 2019-02-14 22:35:52,266 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'TestFIFOCompactionPolicy' with timeout 60000ms 2019-02-14 22:35:52,266 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:52,267 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'TestFIFOCompactionPolicy' starting 'acquire' stage 2019-02-14 22:35:52,267 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2019-02-14 22:35:52,267 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2019-02-14 22:35:52,267 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool13-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:52,267 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool13-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. started... 2019-02-14 22:35:52,267 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool13-thread-1] regionserver.HRegion(2432): Flushing 1/1 column families, memstore=1.25 MB 2019-02-14 22:35:52,272 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool13-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:52,272 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool13-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:52,288 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741847_1023{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f3ca40dc-91f2-42ec-82fe-8f7a3a14736f:NORMAL:127.0.0.1:38317|RBW]]} size 1316785 2019-02-14 22:35:52,689 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool13-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=85, memsize=1.3 M, hasBloomFilter=true, into tmp file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/01700c6968a5474981a43aef16f26aaf 2019-02-14 22:35:52,696 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool13-thread-1] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/01700c6968a5474981a43aef16f26aaf as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/01700c6968a5474981a43aef16f26aaf 2019-02-14 22:35:52,702 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool13-thread-1] regionserver.HStore(1050): Added hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/01700c6968a5474981a43aef16f26aaf, entries=10, sequenceid=85, filesize=1.3 M 2019-02-14 22:35:52,703 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool13-thread-1] regionserver.HRegion(2732): Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 436ms, sequenceid=85, compaction requested=true 2019-02-14 22:35:52,703 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool13-thread-1] regionserver.HRegion(2233): Flush status journal: Acquiring readlock on region at 1550183752267 Running coprocessor pre-flush hooks at 1550183752267 Obtaining lock to block concurrent updates at 1550183752267 Preparing to flush by snapshotting stores in 0b19a16b6042d69e3048c196d2a7c525 at 1550183752267 Finished memstore snapshotting TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525., syncing WAL and waiting on mvcc, flushsize=1311776 at 1550183752268 Flushing stores of TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. at 1550183752270 Flushing f: creating writer at 1550183752270 Flushing f: appending metadata at 1550183752279 Flushing f: closing flushed file at 1550183752279 Flushing f: reopening flushed file at 1550183752697 Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 436ms, sequenceid=85, compaction requested=true at 1550183752703 Running post-flush coprocessor hooks at 1550183752703 Flush successful at 1550183752703 2019-02-14 22:35:52,703 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool13-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:52,703 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2019-02-14 22:35:52,704 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2019-02-14 22:35:52,704 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,704 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'TestFIFOCompactionPolicy' locally acquired 2019-02-14 22:35:52,704 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure (TestFIFOCompactionPolicy) in zk 2019-02-14 22:35:52,706 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,706 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,706 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,706 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:52,706 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:52,707 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,707 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'TestFIFOCompactionPolicy' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2019-02-14 22:35:52,707 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:52,707 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:52,707 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:52,707 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,708 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:52,708 DEBUG [main-EventThread] procedure.Procedure(298): member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure 'TestFIFOCompactionPolicy' on coordinator 2019-02-14 22:35:52,708 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@5033c4cc[Count = 0] remaining members to acquire global barrier 2019-02-14 22:35:52,708 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'TestFIFOCompactionPolicy' starting 'in-barrier' execution. 2019-02-14 22:35:52,708 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,709 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,709 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,709 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,709 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,710 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2019-02-14 22:35:52,710 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'TestFIFOCompactionPolicy' received 'reached' from coordinator. 2019-02-14 22:35:52,710 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'TestFIFOCompactionPolicy' locally completed 2019-02-14 22:35:52,710 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'TestFIFOCompactionPolicy' completed for member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' in zk 2019-02-14 22:35:52,711 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,711 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,711 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:52,711 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:52,711 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'TestFIFOCompactionPolicy' has notified controller of completion 2019-02-14 22:35:52,712 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:52,712 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'TestFIFOCompactionPolicy' completed. 2019-02-14 22:35:52,712 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:52,712 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:52,713 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:52,713 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,713 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:52,713 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:52,714 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,714 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'TestFIFOCompactionPolicy' member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985': 2019-02-14 22:35:52,714 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' released barrier for procedure'TestFIFOCompactionPolicy', counting down latch. Waiting for 0 more 2019-02-14 22:35:52,714 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'TestFIFOCompactionPolicy' execution completed 2019-02-14 22:35:52,714 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2019-02-14 22:35:52,714 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2019-02-14 22:35:52,714 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:TestFIFOCompactionPolicy 2019-02-14 22:35:52,714 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure TestFIFOCompactionPolicyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2019-02-14 22:35:52,718 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,718 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,718 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:52,718 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:52,718 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,718 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,719 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,719 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:52,719 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,719 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:52,719 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:52,719 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:52,719 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:52,719 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:52,719 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,719 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:52,720 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,720 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,720 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:52,720 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:52,720 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,722 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:52,722 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:52,722 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:52,722 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:52,722 DEBUG [main-EventThread] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy because node does not exist (not an error) 2019-02-14 22:35:52,722 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'TestFIFOCompactionPolicy' 2019-02-14 22:35:52,724 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2019-02-14 22:35:52,724 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,724 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:52,724 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:52,724 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:52,724 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:52,725 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:52,724 DEBUG [main] client.HBaseAdmin(4115): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestFIFOCompactionPolicy'' to complete. (max 8571 ms per retry) 2019-02-14 22:35:52,725 DEBUG [main] client.HBaseAdmin(4124): (#1) Sleeping: 100ms while waiting for procedure completion. 2019-02-14 22:35:52,725 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,725 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,725 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:52,725 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,825 DEBUG [main] client.HBaseAdmin(4130): Getting current status of procedure from master... 2019-02-14 22:35:52,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(989): Checking to see if procedure from request:flush-table-proc is done 2019-02-14 22:35:52,855 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(753): Client=apurtell//172.31.47.35 procedure request for: flush-table-proc 2019-02-14 22:35:52,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(155): Procedure TestFIFOCompactionPolicy was in running list but was completed. Accepting new attempt. 2019-02-14 22:35:52,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(177): Submitting procedure TestFIFOCompactionPolicy 2019-02-14 22:35:52,857 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'TestFIFOCompactionPolicy' 2019-02-14 22:35:52,857 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:52,857 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'TestFIFOCompactionPolicy' starting 'acquire' 2019-02-14 22:35:52,857 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'TestFIFOCompactionPolicy', kicking off acquire phase on members. 2019-02-14 22:35:52,858 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,858 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:52,860 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:52,860 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,860 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:52,860 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:52,860 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:52,860 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2019-02-14 22:35:52,860 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:52,860 DEBUG [main-EventThread] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:52,860 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2019-02-14 22:35:52,861 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:52,861 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestFIFOCompactionPolicy 2019-02-14 22:35:52,861 WARN [main-EventThread] procedure.ProcedureMember(142): A completed old subproc TestFIFOCompactionPolicy is still present, removing 2019-02-14 22:35:52,861 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:TestFIFOCompactionPolicy 2019-02-14 22:35:52,861 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'TestFIFOCompactionPolicy' with timeout 60000ms 2019-02-14 22:35:52,861 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:52,862 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'TestFIFOCompactionPolicy' starting 'acquire' stage 2019-02-14 22:35:52,863 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2019-02-14 22:35:52,863 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2019-02-14 22:35:52,863 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool14-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:52,863 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool14-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. started... 2019-02-14 22:35:52,863 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool14-thread-1] regionserver.HRegion(2432): Flushing 1/1 column families, memstore=1.25 MB 2019-02-14 22:35:52,865 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool14-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:52,865 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool14-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:52,876 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741848_1024{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f3ca40dc-91f2-42ec-82fe-8f7a3a14736f:NORMAL:127.0.0.1:38317|RBW]]} size 1316785 2019-02-14 22:35:53,278 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool14-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=99, memsize=1.3 M, hasBloomFilter=true, into tmp file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/e391bc8d7b314d5e889fbac957ede999 2019-02-14 22:35:53,285 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool14-thread-1] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/e391bc8d7b314d5e889fbac957ede999 as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/e391bc8d7b314d5e889fbac957ede999 2019-02-14 22:35:53,292 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool14-thread-1] regionserver.HStore(1050): Added hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/e391bc8d7b314d5e889fbac957ede999, entries=10, sequenceid=99, filesize=1.3 M 2019-02-14 22:35:53,293 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool14-thread-1] regionserver.HRegion(2732): Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 430ms, sequenceid=99, compaction requested=true 2019-02-14 22:35:53,293 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool14-thread-1] regionserver.HRegion(2233): Flush status journal: Acquiring readlock on region at 1550183752863 Running coprocessor pre-flush hooks at 1550183752863 Obtaining lock to block concurrent updates at 1550183752863 Preparing to flush by snapshotting stores in 0b19a16b6042d69e3048c196d2a7c525 at 1550183752863 Finished memstore snapshotting TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525., syncing WAL and waiting on mvcc, flushsize=1311776 at 1550183752863 Flushing stores of TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. at 1550183752864 Flushing f: creating writer at 1550183752864 Flushing f: appending metadata at 1550183752872 Flushing f: closing flushed file at 1550183752873 Flushing f: reopening flushed file at 1550183753286 Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 430ms, sequenceid=99, compaction requested=true at 1550183753293 Running post-flush coprocessor hooks at 1550183753293 Flush successful at 1550183753293 2019-02-14 22:35:53,293 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool14-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:53,294 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2019-02-14 22:35:53,295 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2019-02-14 22:35:53,295 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,295 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'TestFIFOCompactionPolicy' locally acquired 2019-02-14 22:35:53,295 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure (TestFIFOCompactionPolicy) in zk 2019-02-14 22:35:53,297 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,297 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,297 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,297 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:53,297 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:53,297 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,297 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'TestFIFOCompactionPolicy' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2019-02-14 22:35:53,297 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:53,297 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:53,298 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:53,298 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,298 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:53,298 DEBUG [main-EventThread] procedure.Procedure(298): member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure 'TestFIFOCompactionPolicy' on coordinator 2019-02-14 22:35:53,298 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@7bd80d62[Count = 0] remaining members to acquire global barrier 2019-02-14 22:35:53,298 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'TestFIFOCompactionPolicy' starting 'in-barrier' execution. 2019-02-14 22:35:53,298 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,300 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,300 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,300 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,300 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'TestFIFOCompactionPolicy' received 'reached' from coordinator. 2019-02-14 22:35:53,300 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'TestFIFOCompactionPolicy' locally completed 2019-02-14 22:35:53,300 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'TestFIFOCompactionPolicy' completed for member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' in zk 2019-02-14 22:35:53,300 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,300 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2019-02-14 22:35:53,301 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,302 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,302 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:53,302 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:53,302 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'TestFIFOCompactionPolicy' has notified controller of completion 2019-02-14 22:35:53,302 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:53,302 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'TestFIFOCompactionPolicy' completed. 2019-02-14 22:35:53,303 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:53,303 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:53,303 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:53,303 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,303 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:53,304 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:53,304 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,304 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'TestFIFOCompactionPolicy' member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985': 2019-02-14 22:35:53,304 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' released barrier for procedure'TestFIFOCompactionPolicy', counting down latch. Waiting for 0 more 2019-02-14 22:35:53,304 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'TestFIFOCompactionPolicy' execution completed 2019-02-14 22:35:53,305 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2019-02-14 22:35:53,305 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2019-02-14 22:35:53,305 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:TestFIFOCompactionPolicy 2019-02-14 22:35:53,305 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure TestFIFOCompactionPolicyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2019-02-14 22:35:53,306 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,306 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,306 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:53,306 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:53,306 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,307 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,307 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,307 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:53,307 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,307 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:53,307 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:53,307 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:53,307 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:53,307 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,307 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:53,308 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:53,308 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,308 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,308 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:53,308 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:53,309 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,310 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:53,310 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:53,310 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:53,310 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:53,310 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'TestFIFOCompactionPolicy' 2019-02-14 22:35:53,311 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2019-02-14 22:35:53,311 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:53,311 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:53,311 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:53,311 DEBUG [main-EventThread] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy because node does not exist (not an error) 2019-02-14 22:35:53,311 DEBUG [main] client.HBaseAdmin(4115): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestFIFOCompactionPolicy'' to complete. (max 8571 ms per retry) 2019-02-14 22:35:53,312 DEBUG [main] client.HBaseAdmin(4124): (#1) Sleeping: 100ms while waiting for procedure completion. 2019-02-14 22:35:53,312 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,312 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:53,312 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:53,312 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,312 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,312 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,312 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,412 DEBUG [main] client.HBaseAdmin(4130): Getting current status of procedure from master... 2019-02-14 22:35:53,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(989): Checking to see if procedure from request:flush-table-proc is done 2019-02-14 22:35:53,440 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(753): Client=apurtell//172.31.47.35 procedure request for: flush-table-proc 2019-02-14 22:35:53,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(155): Procedure TestFIFOCompactionPolicy was in running list but was completed. Accepting new attempt. 2019-02-14 22:35:53,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(177): Submitting procedure TestFIFOCompactionPolicy 2019-02-14 22:35:53,442 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'TestFIFOCompactionPolicy' 2019-02-14 22:35:53,442 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:53,443 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'TestFIFOCompactionPolicy' starting 'acquire' 2019-02-14 22:35:53,443 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'TestFIFOCompactionPolicy', kicking off acquire phase on members. 2019-02-14 22:35:53,443 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,444 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:53,445 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:53,445 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,445 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:53,445 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:53,446 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,446 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2019-02-14 22:35:53,446 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:53,446 DEBUG [main-EventThread] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,446 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2019-02-14 22:35:53,446 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:53,446 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestFIFOCompactionPolicy 2019-02-14 22:35:53,446 WARN [main-EventThread] procedure.ProcedureMember(142): A completed old subproc TestFIFOCompactionPolicy is still present, removing 2019-02-14 22:35:53,446 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:TestFIFOCompactionPolicy 2019-02-14 22:35:53,447 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'TestFIFOCompactionPolicy' with timeout 60000ms 2019-02-14 22:35:53,447 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:53,447 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'TestFIFOCompactionPolicy' starting 'acquire' stage 2019-02-14 22:35:53,447 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2019-02-14 22:35:53,447 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2019-02-14 22:35:53,447 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool15-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:53,448 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool15-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. started... 2019-02-14 22:35:53,448 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool15-thread-1] regionserver.HRegion(2432): Flushing 1/1 column families, memstore=1.25 MB 2019-02-14 22:35:53,451 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool15-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:53,451 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool15-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:53,465 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741849_1025{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 1316785 2019-02-14 22:35:53,867 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool15-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=113, memsize=1.3 M, hasBloomFilter=true, into tmp file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/6757a48194e54a27b1fffdb1266c807c 2019-02-14 22:35:53,875 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool15-thread-1] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/6757a48194e54a27b1fffdb1266c807c as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/6757a48194e54a27b1fffdb1266c807c 2019-02-14 22:35:53,882 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool15-thread-1] regionserver.HStore(1050): Added hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/6757a48194e54a27b1fffdb1266c807c, entries=10, sequenceid=113, filesize=1.3 M 2019-02-14 22:35:53,883 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool15-thread-1] regionserver.HRegion(2732): Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 435ms, sequenceid=113, compaction requested=true 2019-02-14 22:35:53,883 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool15-thread-1] regionserver.HRegion(2233): Flush status journal: Acquiring readlock on region at 1550183753448 Running coprocessor pre-flush hooks at 1550183753448 Obtaining lock to block concurrent updates at 1550183753448 Preparing to flush by snapshotting stores in 0b19a16b6042d69e3048c196d2a7c525 at 1550183753448 Finished memstore snapshotting TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525., syncing WAL and waiting on mvcc, flushsize=1311776 at 1550183753448 Flushing stores of TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. at 1550183753449 Flushing f: creating writer at 1550183753449 Flushing f: appending metadata at 1550183753461 Flushing f: closing flushed file at 1550183753461 Flushing f: reopening flushed file at 1550183753876 Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 435ms, sequenceid=113, compaction requested=true at 1550183753883 Running post-flush coprocessor hooks at 1550183753883 Flush successful at 1550183753883 2019-02-14 22:35:53,883 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool15-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:53,883 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2019-02-14 22:35:53,884 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2019-02-14 22:35:53,884 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,884 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'TestFIFOCompactionPolicy' locally acquired 2019-02-14 22:35:53,884 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure (TestFIFOCompactionPolicy) in zk 2019-02-14 22:35:53,886 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,886 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,886 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,886 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:53,887 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:53,887 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,887 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'TestFIFOCompactionPolicy' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2019-02-14 22:35:53,887 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:53,887 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:53,887 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:53,887 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,888 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:53,888 DEBUG [main-EventThread] procedure.Procedure(298): member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure 'TestFIFOCompactionPolicy' on coordinator 2019-02-14 22:35:53,888 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@435232b8[Count = 0] remaining members to acquire global barrier 2019-02-14 22:35:53,888 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'TestFIFOCompactionPolicy' starting 'in-barrier' execution. 2019-02-14 22:35:53,888 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,889 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,889 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,889 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,890 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'TestFIFOCompactionPolicy' received 'reached' from coordinator. 2019-02-14 22:35:53,890 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'TestFIFOCompactionPolicy' locally completed 2019-02-14 22:35:53,890 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'TestFIFOCompactionPolicy' completed for member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' in zk 2019-02-14 22:35:53,890 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,890 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2019-02-14 22:35:53,891 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,891 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'TestFIFOCompactionPolicy' has notified controller of completion 2019-02-14 22:35:53,891 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:53,891 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,891 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:53,892 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:53,891 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'TestFIFOCompactionPolicy' completed. 2019-02-14 22:35:53,893 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:53,893 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:53,893 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:53,893 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,893 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:53,894 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:53,894 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,894 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'TestFIFOCompactionPolicy' member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985': 2019-02-14 22:35:53,894 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' released barrier for procedure'TestFIFOCompactionPolicy', counting down latch. Waiting for 0 more 2019-02-14 22:35:53,894 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'TestFIFOCompactionPolicy' execution completed 2019-02-14 22:35:53,894 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2019-02-14 22:35:53,894 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2019-02-14 22:35:53,894 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:TestFIFOCompactionPolicy 2019-02-14 22:35:53,894 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure TestFIFOCompactionPolicyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2019-02-14 22:35:53,896 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,896 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,896 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:53,896 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:53,896 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,896 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,896 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,897 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:53,897 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,897 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:53,897 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:53,897 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:53,897 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:53,897 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:53,897 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,897 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:53,897 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,898 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,898 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:53,898 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:53,898 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,900 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:53,900 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:53,900 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:53,900 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:53,900 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'TestFIFOCompactionPolicy' 2019-02-14 22:35:53,901 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:53,901 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:53,901 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:53,900 DEBUG [main-EventThread] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy because node does not exist (not an error) 2019-02-14 22:35:53,901 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,901 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2019-02-14 22:35:53,901 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:53,901 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:53,901 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:53,901 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,901 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:53,901 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:53,901 DEBUG [main] client.HBaseAdmin(4115): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestFIFOCompactionPolicy'' to complete. (max 8571 ms per retry) 2019-02-14 22:35:53,901 DEBUG [main] client.HBaseAdmin(4124): (#1) Sleeping: 100ms while waiting for procedure completion. 2019-02-14 22:35:54,002 DEBUG [main] client.HBaseAdmin(4130): Getting current status of procedure from master... 2019-02-14 22:35:54,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(989): Checking to see if procedure from request:flush-table-proc is done 2019-02-14 22:35:54,028 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(753): Client=apurtell//172.31.47.35 procedure request for: flush-table-proc 2019-02-14 22:35:54,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(155): Procedure TestFIFOCompactionPolicy was in running list but was completed. Accepting new attempt. 2019-02-14 22:35:54,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(177): Submitting procedure TestFIFOCompactionPolicy 2019-02-14 22:35:54,031 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'TestFIFOCompactionPolicy' 2019-02-14 22:35:54,031 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:54,032 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'TestFIFOCompactionPolicy' starting 'acquire' 2019-02-14 22:35:54,032 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'TestFIFOCompactionPolicy', kicking off acquire phase on members. 2019-02-14 22:35:54,032 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:54,032 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:54,033 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:54,033 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,033 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:54,034 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:54,034 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,034 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2019-02-14 22:35:54,034 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:54,034 DEBUG [main-EventThread] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:54,034 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2019-02-14 22:35:54,034 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:54,034 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestFIFOCompactionPolicy 2019-02-14 22:35:54,035 WARN [main-EventThread] procedure.ProcedureMember(142): A completed old subproc TestFIFOCompactionPolicy is still present, removing 2019-02-14 22:35:54,035 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:TestFIFOCompactionPolicy 2019-02-14 22:35:54,035 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'TestFIFOCompactionPolicy' with timeout 60000ms 2019-02-14 22:35:54,035 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:54,035 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'TestFIFOCompactionPolicy' starting 'acquire' stage 2019-02-14 22:35:54,035 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2019-02-14 22:35:54,035 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2019-02-14 22:35:54,036 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool16-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:54,036 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool16-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. started... 2019-02-14 22:35:54,036 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool16-thread-1] regionserver.HRegion(2432): Flushing 1/1 column families, memstore=1.25 MB 2019-02-14 22:35:54,038 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool16-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:54,038 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool16-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:54,050 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741850_1026{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 1316785 2019-02-14 22:35:54,176 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2019-02-14 22:35:54,451 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool16-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=127, memsize=1.3 M, hasBloomFilter=true, into tmp file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/2d2bf9b738554a288d421a80cde645a9 2019-02-14 22:35:54,458 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool16-thread-1] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/2d2bf9b738554a288d421a80cde645a9 as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/2d2bf9b738554a288d421a80cde645a9 2019-02-14 22:35:54,463 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool16-thread-1] regionserver.HStore(1050): Added hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/2d2bf9b738554a288d421a80cde645a9, entries=10, sequenceid=127, filesize=1.3 M 2019-02-14 22:35:54,464 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool16-thread-1] regionserver.HRegion(2732): Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 428ms, sequenceid=127, compaction requested=true 2019-02-14 22:35:54,464 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool16-thread-1] regionserver.HRegion(2233): Flush status journal: Acquiring readlock on region at 1550183754036 Running coprocessor pre-flush hooks at 1550183754036 Obtaining lock to block concurrent updates at 1550183754036 Preparing to flush by snapshotting stores in 0b19a16b6042d69e3048c196d2a7c525 at 1550183754036 Finished memstore snapshotting TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525., syncing WAL and waiting on mvcc, flushsize=1311776 at 1550183754036 Flushing stores of TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. at 1550183754036 Flushing f: creating writer at 1550183754036 Flushing f: appending metadata at 1550183754045 Flushing f: closing flushed file at 1550183754045 Flushing f: reopening flushed file at 1550183754459 Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 428ms, sequenceid=127, compaction requested=true at 1550183754464 Running post-flush coprocessor hooks at 1550183754464 Flush successful at 1550183754464 2019-02-14 22:35:54,464 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool16-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:54,465 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2019-02-14 22:35:54,466 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2019-02-14 22:35:54,466 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,466 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'TestFIFOCompactionPolicy' locally acquired 2019-02-14 22:35:54,466 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure (TestFIFOCompactionPolicy) in zk 2019-02-14 22:35:54,468 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,468 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:54,468 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,468 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:54,468 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:54,468 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:54,468 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'TestFIFOCompactionPolicy' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2019-02-14 22:35:54,468 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:54,469 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:54,469 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:54,469 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,469 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:54,470 DEBUG [main-EventThread] procedure.Procedure(298): member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure 'TestFIFOCompactionPolicy' on coordinator 2019-02-14 22:35:54,470 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@532f0f62[Count = 0] remaining members to acquire global barrier 2019-02-14 22:35:54,470 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'TestFIFOCompactionPolicy' starting 'in-barrier' execution. 2019-02-14 22:35:54,470 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:54,471 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:54,471 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:54,471 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:54,471 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'TestFIFOCompactionPolicy' received 'reached' from coordinator. 2019-02-14 22:35:54,471 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'TestFIFOCompactionPolicy' locally completed 2019-02-14 22:35:54,471 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'TestFIFOCompactionPolicy' completed for member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' in zk 2019-02-14 22:35:54,471 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,472 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2019-02-14 22:35:54,473 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,473 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,473 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:54,473 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:54,473 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'TestFIFOCompactionPolicy' has notified controller of completion 2019-02-14 22:35:54,473 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:54,473 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'TestFIFOCompactionPolicy' completed. 2019-02-14 22:35:54,474 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:54,474 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:54,474 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:54,475 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,475 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:54,475 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:54,475 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,476 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'TestFIFOCompactionPolicy' member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985': 2019-02-14 22:35:54,476 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' released barrier for procedure'TestFIFOCompactionPolicy', counting down latch. Waiting for 0 more 2019-02-14 22:35:54,476 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'TestFIFOCompactionPolicy' execution completed 2019-02-14 22:35:54,476 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2019-02-14 22:35:54,476 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2019-02-14 22:35:54,476 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:TestFIFOCompactionPolicy 2019-02-14 22:35:54,476 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure TestFIFOCompactionPolicyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2019-02-14 22:35:54,477 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:54,478 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:54,478 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:54,478 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:54,478 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:54,478 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:54,478 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:54,478 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:54,478 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:54,478 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:54,478 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,478 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:54,478 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:54,478 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:54,478 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:54,479 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:54,479 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,479 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,479 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:54,479 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:54,480 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,481 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:54,481 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:54,481 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:54,481 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:54,481 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'TestFIFOCompactionPolicy' 2019-02-14 22:35:54,482 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2019-02-14 22:35:54,482 DEBUG [main-EventThread] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy because node does not exist (not an error) 2019-02-14 22:35:54,482 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,481 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:54,482 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:54,483 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:54,482 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:54,483 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:54,483 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,483 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:54,483 DEBUG [main] client.HBaseAdmin(4115): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestFIFOCompactionPolicy'' to complete. (max 8571 ms per retry) 2019-02-14 22:35:54,483 DEBUG [main] client.HBaseAdmin(4124): (#1) Sleeping: 100ms while waiting for procedure completion. 2019-02-14 22:35:54,483 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:54,483 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:54,583 DEBUG [main] client.HBaseAdmin(4130): Getting current status of procedure from master... 2019-02-14 22:35:54,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(989): Checking to see if procedure from request:flush-table-proc is done 2019-02-14 22:35:54,609 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(753): Client=apurtell//172.31.47.35 procedure request for: flush-table-proc 2019-02-14 22:35:54,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(155): Procedure TestFIFOCompactionPolicy was in running list but was completed. Accepting new attempt. 2019-02-14 22:35:54,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] procedure.ProcedureCoordinator(177): Submitting procedure TestFIFOCompactionPolicy 2019-02-14 22:35:54,611 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(196): Starting procedure 'TestFIFOCompactionPolicy' 2019-02-14 22:35:54,611 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:54,611 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(204): Procedure 'TestFIFOCompactionPolicy' starting 'acquire' 2019-02-14 22:35:54,611 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(247): Starting procedure 'TestFIFOCompactionPolicy', kicking off acquire phase on members. 2019-02-14 22:35:54,612 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:54,612 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(94): Creating acquire znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:54,614 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(102): Watching for acquire node:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,614 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:54,614 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:54,614 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:54,614 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:54,614 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(208): Waiting for all members to 'acquire' 2019-02-14 22:35:54,614 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(184): Found procedure znode: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:54,614 DEBUG [main-EventThread] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:54,615 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(210): start proc data length is 4 2019-02-14 22:35:54,615 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(212): Found data for znode:/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:54,615 DEBUG [main-EventThread] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestFIFOCompactionPolicy 2019-02-14 22:35:54,615 WARN [main-EventThread] procedure.ProcedureMember(142): A completed old subproc TestFIFOCompactionPolicy is still present, removing 2019-02-14 22:35:54,615 DEBUG [main-EventThread] procedure.ProcedureMember(149): Submitting new Subprocedure:TestFIFOCompactionPolicy 2019-02-14 22:35:54,615 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(158): Starting subprocedure 'TestFIFOCompactionPolicy' with timeout 60000ms 2019-02-14 22:35:54,615 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(108): Scheduling process timer to run in: 60000 ms 2019-02-14 22:35:54,616 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(166): Subprocedure 'TestFIFOCompactionPolicy' starting 'acquire' stage 2019-02-14 22:35:54,616 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.FlushTableSubprocedure(100): Flush region tasks submitted for 1 regions 2019-02-14 22:35:54,616 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool17-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(64): Starting region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:54,616 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(243): Waiting for local region flush to finish. 2019-02-14 22:35:54,616 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool17-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(67): Flush region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. started... 2019-02-14 22:35:54,616 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool17-thread-1] regionserver.HRegion(2432): Flushing 1/1 column families, memstore=1.25 MB 2019-02-14 22:35:54,618 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool17-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:54,618 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool17-thread-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:54,630 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741851_1027{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 1316795 2019-02-14 22:35:55,031 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool17-thread-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=141, memsize=1.3 M, hasBloomFilter=true, into tmp file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/3e3376540ae34b52b351b80a2d1ea6e6 2019-02-14 22:35:55,039 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool17-thread-1] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/3e3376540ae34b52b351b80a2d1ea6e6 as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/3e3376540ae34b52b351b80a2d1ea6e6 2019-02-14 22:35:55,044 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool17-thread-1] regionserver.HStore(1050): Added hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/3e3376540ae34b52b351b80a2d1ea6e6, entries=10, sequenceid=141, filesize=1.3 M 2019-02-14 22:35:55,045 INFO [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool17-thread-1] regionserver.HRegion(2732): Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 429ms, sequenceid=141, compaction requested=true 2019-02-14 22:35:55,045 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool17-thread-1] regionserver.HRegion(2233): Flush status journal: Acquiring readlock on region at 1550183754616 Running coprocessor pre-flush hooks at 1550183754616 Obtaining lock to block concurrent updates at 1550183754616 Preparing to flush by snapshotting stores in 0b19a16b6042d69e3048c196d2a7c525 at 1550183754616 Finished memstore snapshotting TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525., syncing WAL and waiting on mvcc, flushsize=1311776 at 1550183754616 Flushing stores of TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. at 1550183754617 Flushing f: creating writer at 1550183754617 Flushing f: appending metadata at 1550183754627 Flushing f: closing flushed file at 1550183754627 Flushing f: reopening flushed file at 1550183755040 Finished memstore flush of ~1.25 MB/1311776, currentsize=0 B/0 for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. in 429ms, sequenceid=141, compaction requested=true at 1550183755045 Running post-flush coprocessor hooks at 1550183755045 Flush successful at 1550183755045 2019-02-14 22:35:55,046 DEBUG [rs(ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985)-flush-proc-pool17-thread-1] flush.FlushTableSubprocedure$RegionFlushTask(71): Closing region operation on TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:55,046 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(254): Completed 1/1 local region flush tasks. 2019-02-14 22:35:55,046 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(256): Completed 1 local region flush tasks. 2019-02-14 22:35:55,046 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(289): cancelling 0 flush region tasks ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,046 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(168): Subprocedure 'TestFIFOCompactionPolicy' locally acquired 2019-02-14 22:35:55,047 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(241): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure (TestFIFOCompactionPolicy) in zk 2019-02-14 22:35:55,050 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,050 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(249): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:55,050 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,050 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:55,050 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:55,050 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:55,050 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(173): Subprocedure 'TestFIFOCompactionPolicy' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2019-02-14 22:35:55,050 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:55,051 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:55,051 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:55,051 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,051 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:55,051 DEBUG [main-EventThread] procedure.Procedure(298): member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' joining acquired barrier for procedure 'TestFIFOCompactionPolicy' on coordinator 2019-02-14 22:35:55,051 DEBUG [main-EventThread] procedure.Procedure(307): Waiting on: java.util.concurrent.CountDownLatch@1df4b46f[Count = 0] remaining members to acquire global barrier 2019-02-14 22:35:55,052 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(212): Procedure 'TestFIFOCompactionPolicy' starting 'in-barrier' execution. 2019-02-14 22:35:55,052 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(118): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:55,053 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:55,053 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:55,053 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(130): Received reached global barrier:/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:55,054 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(187): Subprocedure 'TestFIFOCompactionPolicy' received 'reached' from coordinator. 2019-02-14 22:35:55,054 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(189): Subprocedure 'TestFIFOCompactionPolicy' locally completed 2019-02-14 22:35:55,054 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(265): Marking procedure 'TestFIFOCompactionPolicy' completed for member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' in zk 2019-02-14 22:35:55,054 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,054 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(216): Waiting for all members to 'release' 2019-02-14 22:35:55,055 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,056 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,056 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:55,056 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:55,055 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(194): Subprocedure 'TestFIFOCompactionPolicy' has notified controller of completion 2019-02-14 22:35:55,056 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:55,056 DEBUG [member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' subprocedure-pool2-thread-1] procedure.Subprocedure(219): Subprocedure 'TestFIFOCompactionPolicy' completed. 2019-02-14 22:35:55,056 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:55,057 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:55,057 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:55,057 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,058 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:55,058 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:55,058 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,058 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(221): Finished data from procedure 'TestFIFOCompactionPolicy' member 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985': 2019-02-14 22:35:55,058 DEBUG [main-EventThread] procedure.Procedure(329): Member: 'ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985' released barrier for procedure'TestFIFOCompactionPolicy', counting down latch. Waiting for 0 more 2019-02-14 22:35:55,058 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(221): Procedure 'TestFIFOCompactionPolicy' execution completed 2019-02-14 22:35:55,058 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(230): Running finish phase. 2019-02-14 22:35:55,058 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.Procedure(281): Finished coordinator procedure - removing self from list of running procedures 2019-02-14 22:35:55,059 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureCoordinatorRpcs(165): Attempting to clean out zk node for op:TestFIFOCompactionPolicy 2019-02-14 22:35:55,059 INFO [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] procedure.ZKProcedureUtil(285): Clearing all znodes for procedure TestFIFOCompactionPolicyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2019-02-14 22:35:55,060 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:55,060 DEBUG [main-EventThread] procedure.ZKProcedureCoordinatorRpcs$1(196): Node created: /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:55,060 DEBUG [main-EventThread] procedure.ZKProcedureUtil(244): Current zk system: 2019-02-14 22:35:55,060 DEBUG [main-EventThread] procedure.ZKProcedureUtil(246): |-/hbase/flush-table-proc 2019-02-14 22:35:55,060 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:55,060 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(79): Received created event:/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:55,061 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:55,061 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-abort 2019-02-14 22:35:55,061 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:55,061 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:55,061 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:55,061 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:55,061 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,061 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(313): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:55,061 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-acquired 2019-02-14 22:35:55,061 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:55,062 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,062 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] zookeeper.ZKUtil(374): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,062 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-reached 2019-02-14 22:35:55,062 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |----TestFIFOCompactionPolicy 2019-02-14 22:35:55,062 DEBUG [main-EventThread] procedure.ZKProcedureUtil(263): |-------ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,064 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2019-02-14 22:35:55,064 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(106): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2019-02-14 22:35:55,064 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2019-02-14 22:35:55,064 DEBUG [(ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583)-proc-coordinator-pool1-thread-1] errorhandling.TimeoutExceptionInjector(88): Marking timer as complete - no error notifications will be received for this timer. 2019-02-14 22:35:55,064 DEBUG [main-EventThread] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestFIFOCompactionPolicy because node does not exist (not an error) 2019-02-14 22:35:55,064 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,065 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:55,064 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(169): Done waiting - exec procedure flush-table-proc for 'TestFIFOCompactionPolicy' 2019-02-14 22:35:55,065 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] flush.MasterFlushTableProcedureManager(171): Master flush table procedure is successful! 2019-02-14 22:35:55,065 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2019-02-14 22:35:55,065 INFO [main-EventThread] procedure.ZKProcedureMemberRpcs$1(109): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2019-02-14 22:35:55,065 DEBUG [main-EventThread] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2019-02-14 22:35:55,065 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestFIFOCompactionPolicy 2019-02-14 22:35:55,065 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,065 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:55,065 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestFIFOCompactionPolicy 2019-02-14 22:35:55,065 DEBUG [main] client.HBaseAdmin(4115): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestFIFOCompactionPolicy'' to complete. (max 8571 ms per retry) 2019-02-14 22:35:55,065 DEBUG [main] client.HBaseAdmin(4124): (#1) Sleeping: 100ms while waiting for procedure completion. 2019-02-14 22:35:55,065 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestFIFOCompactionPolicy 2019-02-14 22:35:55,166 DEBUG [main] client.HBaseAdmin(4130): Getting current status of procedure from master... 2019-02-14 22:35:55,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] master.MasterRpcServices(989): Checking to see if procedure from request:flush-table-proc is done 2019-02-14 22:35:55,169 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hbase-admin-on-hconnection-0x39e67516 connecting to ZooKeeper ensemble=localhost:50587 2019-02-14 22:35:55,172 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.RSRpcServices(1480): Compacting TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:55,172 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.DefaultStoreFileManager(204): Found an expired store file: hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/9d1822e1d3a447c3a6fdc07c8d3a3c7b whose maxTimeStamp is 1550183749652, which is below 1550183764182 2019-02-14 22:35:55,173 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.DefaultStoreFileManager(204): Found an expired store file: hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/51309c87436c439aa1a005af85f79252 whose maxTimeStamp is 1550183750854, which is below 1550183764182 2019-02-14 22:35:55,173 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.DefaultStoreFileManager(204): Found an expired store file: hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/6c664385246141a4bb1aa0301536b8f5 whose maxTimeStamp is 1550183752460, which is below 1550183764182 2019-02-14 22:35:55,173 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): hbase-admin-on-hconnection-0x39e675160x0, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2019-02-14 22:35:55,173 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.DefaultStoreFileManager(204): Found an expired store file: hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/ae7b89c41de44052a04c52f21a98c14c whose maxTimeStamp is 1550183754056, which is below 1550183764182 2019-02-14 22:35:55,173 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(678): hbase-admin-on-hconnection-0x39e67516-0x168ee26a5f9000c connected 2019-02-14 22:35:55,174 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.DefaultStoreFileManager(204): Found an expired store file: hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/34bb22bdc7b247b9b10b363517de408e whose maxTimeStamp is 1550183755658, which is below 1550183764182 2019-02-14 22:35:55,174 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.DefaultStoreFileManager(204): Found an expired store file: hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/01700c6968a5474981a43aef16f26aaf whose maxTimeStamp is 1550183757254, which is below 1550183764182 2019-02-14 22:35:55,174 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.DefaultStoreFileManager(204): Found an expired store file: hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/e391bc8d7b314d5e889fbac957ede999 whose maxTimeStamp is 1550183758855, which is below 1550183764182 2019-02-14 22:35:55,174 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.DefaultStoreFileManager(204): Found an expired store file: hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/6757a48194e54a27b1fffdb1266c807c whose maxTimeStamp is 1550183760442, which is below 1550183764182 2019-02-14 22:35:55,174 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.DefaultStoreFileManager(204): Found an expired store file: hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/2d2bf9b738554a288d421a80cde645a9 whose maxTimeStamp is 1550183762031, which is below 1550183764182 2019-02-14 22:35:55,175 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.HStore(1938): Completing compaction... 2019-02-14 22:35:55,175 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.HStore(1857): Completed removal of 9 unnecessary (expired) file(s) in f of TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.; total size for store is 1.3 M 2019-02-14 22:35:55,175 WARN [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] compactions.FIFOCompactionPolicy(63): Major compaction is not supported for FIFO compaction policy. Ignore the flag. 2019-02-14 22:35:55,175 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.HStore(1812): 0b19a16b6042d69e3048c196d2a7c525 - f: Initiating minor compaction 2019-02-14 22:35:55,175 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=39948] regionserver.CompactSplitThread(400): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@45fb4f05; Because: User-triggered major compaction; compaction_queue=(0:1), split_queue=0, merge_queue=0 2019-02-14 22:35:55,175 INFO [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.HRegion(2097): Starting compaction on f in region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:55,176 INFO [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.HStore(1355): Starting compaction of 1 file(s) in f of TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. into tmpdir=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp, totalSize=1.3 M 2019-02-14 22:35:55,176 DEBUG [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] compactions.Compactor(197): Compacting hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/3e3376540ae34b52b351b80a2d1ea6e6, keycount=10, bloomtype=ROW, size=1.3 M, encoding=NONE, compression=NONE, seqNum=141 2019-02-14 22:35:55,178 INFO [main] hbase.Waiter(189): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2019-02-14 22:35:55,187 DEBUG [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:55,187 DEBUG [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f 2019-02-14 22:35:55,189 INFO [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] throttle.PressureAwareThroughputController(153): TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.#f#compaction#14 average throughput is NaN MB/sec, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/sec 2019-02-14 22:35:55,194 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741852_1028{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 4527 2019-02-14 22:35:55,199 INFO [main] hbase.ResourceChecker(171): after: regionserver.compactions.TestFIFOCompactionPolicy#testPurgeExpiredFiles Thread=305 (was 249) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t129 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x5774ef9b-shared--pool8-t24 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t79 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t105 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t117 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t91 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t88 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t126 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t122 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t127 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t75 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t111 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t112 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t114 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t90 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t85 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t110 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t102 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t107 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t130 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t120 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t76 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t84 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: AM.ZK.Worker-pool5-t17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t71 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t73 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x5774ef9b-shared--pool8-t22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t116 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t119 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: AM.ZK.Worker-pool5-t16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t104 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t93 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t113 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t124 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t72 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t108 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t118 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t80 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t89 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t77 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t96 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t132 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t131 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x5774ef9b-shared--pool8-t21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t121 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t74 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x5774ef9b-shared--pool8-t25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1043254411_1 at /127.0.0.1:37826 [Sending block BP-1808774508-172.31.47.35-1550183734971:blk_1073741851_1027] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t94 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: AM.ZK.Worker-pool5-t14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t87 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: AM.ZK.Worker-pool5-t15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: AM.ZK.Worker-pool5-t18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t92 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t78 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t101 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t83 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t128 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t103 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t81 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t86 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t95 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_OPEN_REGION-ip-172-31-47-35:39948-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t123 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t115 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x5774ef9b-shared--pool8-t23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t98 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t100 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t109 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t82 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t99 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t97 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t125 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x39e67516-shared--pool19-t106 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=564 (was 564), MaxFileDescriptor=65536 (was 65536), SystemLoadAverage=243 (was 270), ProcessCount=429 (was 429), AvailableMemoryMB=8570 (was 8682), ConnectionCount=1 (was 1) 2019-02-14 22:35:55,219 INFO [main] hbase.ResourceChecker(147): before: regionserver.compactions.TestFIFOCompactionPolicy#testSanityCheckBlockingStoreFiles Thread=306, OpenFileDescriptor=564, MaxFileDescriptor=65536, SystemLoadAverage=243, ProcessCount=429, AvailableMemoryMB=8568, ConnectionCount=1 2019-02-14 22:35:55,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925] ipc.CallRunner(130): RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40925: callId: 225 service: MasterService methodName: CreateTable size: 626 connection: 172.31.47.35:42176 deadline: 1550183815220 org.apache.hadoop.hbase.DoNotRetryIOException: blocking file count 'hbase.hstore.blockingStoreFiles' 10 is below recommended minimum of 1000 Set hbase.table.sanity.checks to false at conf or table descriptor if you want to bypass sanity checks at org.apache.hadoop.hbase.master.HMaster.warnOrThrowExceptionForFailure(HMaster.java:1993) at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1859) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1741) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:482) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:63400) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277) Caused by: java.io.IOException: blocking file count 'hbase.hstore.blockingStoreFiles' 10 is below recommended minimum of 1000 at org.apache.hadoop.hbase.master.HMaster.checkCompactionPolicy(HMaster.java:1984) at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1857) ... 7 more 2019-02-14 22:35:55,241 INFO [main] hbase.ResourceChecker(171): after: regionserver.compactions.TestFIFOCompactionPolicy#testSanityCheckBlockingStoreFiles Thread=306 (was 306), OpenFileDescriptor=564 (was 564), MaxFileDescriptor=65536 (was 65536), SystemLoadAverage=243 (was 243), ProcessCount=429 (was 429), AvailableMemoryMB=8568 (was 8568), ConnectionCount=1 (was 1) 2019-02-14 22:35:55,241 INFO [main] hbase.HBaseTestingUtility(1172): Shutting down minicluster 2019-02-14 22:35:55,241 INFO [main] client.ConnectionManager$HConnectionImplementation(2251): Closing master protocol: MasterService 2019-02-14 22:35:55,243 INFO [main] client.ConnectionManager$HConnectionImplementation(1774): Closing zookeeper sessionid=0x168ee26a5f90007 2019-02-14 22:35:55,245 DEBUG [main] ipc.AbstractRpcClient(483): Stopping rpc client 2019-02-14 22:35:55,245 DEBUG [RpcServer.reader=2,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=40925] ipc.RpcServer$Listener(971): RpcServer.listener,port=40925: DISCONNECTING client 172.31.47.35:42176 because read count=-1. Number of active connections: 2 2019-02-14 22:35:55,246 DEBUG [main] util.JVMClusterUtil(246): Shutting down HBase Cluster 2019-02-14 22:35:55,246 DEBUG [RpcServer.reader=1,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Listener(971): RpcServer.listener,port=39948: DISCONNECTING client 172.31.47.35:51788 because read count=-1. Number of active connections: 5 2019-02-14 22:35:55,246 DEBUG [RpcServer.reader=1,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Listener(971): RpcServer.listener,port=39948: DISCONNECTING client 172.31.47.35:51830 because read count=-1. Number of active connections: 4 2019-02-14 22:35:55,250 INFO [main] regionserver.HRegionServer(2027): STOPPED: Cluster shutdown requested 2019-02-14 22:35:55,251 INFO [M:0;ip-172-31-47-35:40925] procedure2.ProcedureExecutor(519): Stopping the procedure executor 2019-02-14 22:35:55,251 INFO [M:0;ip-172-31-47-35:40925] wal.WALProcedureStore(251): Stopping the WAL Procedure Store 2019-02-14 22:35:55,253 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2019-02-14 22:35:55,253 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2019-02-14 22:35:55,253 DEBUG [main-EventThread] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2019-02-14 22:35:55,255 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2019-02-14 22:35:55,255 DEBUG [main-EventThread] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2019-02-14 22:35:55,255 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2019-02-14 22:35:55,255 INFO [main] regionserver.HRegionServer(2027): STOPPED: Shutdown requested 2019-02-14 22:35:55,255 DEBUG [main-EventThread] zookeeper.ZKUtil(376): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2019-02-14 22:35:55,255 INFO [RS:0;ip-172-31-47-35:39948] regionserver.SplitLogWorker(170): Sending interrupt to stop the worker thread 2019-02-14 22:35:55,255 DEBUG [main-EventThread] zookeeper.ZKUtil(376): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2019-02-14 22:35:55,255 INFO [SplitLogWorker-ip-172-31-47-35:39948] regionserver.SplitLogWorker(152): SplitLogWorker interrupted. Exiting. 2019-02-14 22:35:55,256 INFO [RS:0;ip-172-31-47-35:39948] regionserver.HeapMemoryManager(216): Stopping HeapMemoryTuner chore. 2019-02-14 22:35:55,256 INFO [SplitLogWorker-ip-172-31-47-35:39948] regionserver.SplitLogWorker(161): SplitLogWorker ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 exiting 2019-02-14 22:35:55,257 INFO [RS:0;ip-172-31-47-35:39948] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2019-02-14 22:35:55,257 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(285): MemStoreFlusher.1 exiting 2019-02-14 22:35:55,257 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(285): MemStoreFlusher.0 exiting 2019-02-14 22:35:55,257 INFO [RS:0;ip-172-31-47-35:39948] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2019-02-14 22:35:55,261 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-0] handler.CloseRegionHandler(119): Processing close of TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:55,261 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-1] handler.CloseRegionHandler(119): Processing close of hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5. 2019-02-14 22:35:55,261 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-0] regionserver.HRegion(1526): Closing TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.: disabling compactions & flushes 2019-02-14 22:35:55,261 INFO [RS:0;ip-172-31-47-35:39948] regionserver.HRegionServer(1124): stopping server ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,261 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-2] handler.CloseRegionHandler(119): Processing close of testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:55,261 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-0] regionserver.HRegion(1678): waiting for 1 compactions to complete for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:55,261 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-1] regionserver.HRegion(1526): Closing hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5.: disabling compactions & flushes 2019-02-14 22:35:55,262 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-2] regionserver.HRegion(1526): Closing testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0.: disabling compactions & flushes 2019-02-14 22:35:55,262 DEBUG [RS:0;ip-172-31-47-35:39948] zookeeper.MetaTableLocator(623): Stopping MetaTableLocator 2019-02-14 22:35:55,262 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-2] regionserver.HRegion(1553): Updates disabled for region testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:55,262 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-1] regionserver.HRegion(1553): Updates disabled for region hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5. 2019-02-14 22:35:55,262 INFO [RS:0;ip-172-31-47-35:39948] client.ConnectionManager$HConnectionImplementation(1774): Closing zookeeper sessionid=0x168ee26a5f90003 2019-02-14 22:35:55,262 INFO [RS_CLOSE_REGION-ip-172-31-47-35:39948-1] regionserver.HRegion(2432): Flushing 1/1 column families, memstore=248 B 2019-02-14 22:35:55,264 DEBUG [StoreCloserThread-testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0.-1] regionserver.HStore(2878): Moving the files [hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/374b7b8dc7314f24a42098a19473b3bc, hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/2456003b842340a8a586d3721c8cc5e1] to archive 2019-02-14 22:35:55,264 DEBUG [RS:0;ip-172-31-47-35:39948] ipc.AbstractRpcClient(483): Stopping rpc client 2019-02-14 22:35:55,265 INFO [RS:0;ip-172-31-47-35:39948] regionserver.CompactSplitThread(445): Waiting for Split Thread to finish... 2019-02-14 22:35:55,265 INFO [RS:0;ip-172-31-47-35:39948] regionserver.CompactSplitThread(445): Waiting for Merge Thread to finish... 2019-02-14 22:35:55,265 INFO [RS:0;ip-172-31-47-35:39948] regionserver.CompactSplitThread(445): Waiting for Large Compaction Thread to finish... 2019-02-14 22:35:55,267 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741831_1007{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f3ca40dc-91f2-42ec-82fe-8f7a3a14736f:NORMAL:127.0.0.1:38317|RBW]]} size 15506 2019-02-14 22:35:55,268 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/namespace/c71b20fd2578c13df17ccd07b9ab4dd5/.tmp/info 2019-02-14 22:35:55,268 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-1] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/namespace/c71b20fd2578c13df17ccd07b9ab4dd5/.tmp/info 2019-02-14 22:35:55,269 DEBUG [StoreCloserThread-testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0.-1] backup.HFileArchiver(235): Archiving compacted store files. 2019-02-14 22:35:55,275 DEBUG [StoreCloserThread-testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0.-1] backup.HFileArchiver(427): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/374b7b8dc7314f24a42098a19473b3bc, to hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/archive/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/374b7b8dc7314f24a42098a19473b3bc 2019-02-14 22:35:55,277 DEBUG [StoreCloserThread-testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0.-1] backup.HFileArchiver(427): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/2456003b842340a8a586d3721c8cc5e1, to hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/archive/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/f/2456003b842340a8a586d3721c8cc5e1 2019-02-14 22:35:55,279 INFO [StoreCloserThread-testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0.-1] regionserver.HStore(954): Closed f 2019-02-14 22:35:55,280 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741853_1029{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 4963 2019-02-14 22:35:55,284 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-2] wal.WALSplitter(751): Wrote region seqId=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/default/testFIFOCompactionPolicyExpiredEmptyHFiles/39958fd836f676bdcfb70cc61f4ef4c0/recovered.edits/21.seqid to file, newSeqId=21, maxSeqId=2 2019-02-14 22:35:55,285 INFO [RS_CLOSE_REGION-ip-172-31-47-35:39948-2] regionserver.HRegion(1660): Closed testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:55,285 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-2] handler.CloseRegionHandler(164): Closed testFIFOCompactionPolicyExpiredEmptyHFiles,,1550183743520.39958fd836f676bdcfb70cc61f4ef4c0. 2019-02-14 22:35:55,383 INFO [regionserver/ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:0.logRoller] regionserver.LogRoller(201): LogRoller exiting. 2019-02-14 22:35:55,384 INFO [regionserver/ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:0.leaseChecker] regionserver.Leases(147): regionserver/ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:0.leaseChecker closing leases 2019-02-14 22:35:55,384 INFO [regionserver/ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:0.leaseChecker] regionserver.Leases(150): regionserver/ip-172-31-47-35.us-west-2.compute.internal/172.31.47.35:0.leaseChecker closed leases 2019-02-14 22:35:55,394 INFO [ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985_ChoreService_1] hbase.ScheduledChore(183): Chore: ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985-MemstoreFlusherChore was stopped 2019-02-14 22:35:55,394 INFO [ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985_ChoreService_1] hbase.ScheduledChore(183): Chore: CompactionChecker was stopped 2019-02-14 22:35:55,601 INFO [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.StoreFile(495): HFile Bloom filter type for 131cd200dfb44b5a8bae5fd317fc4129: NONE, but ROW specified in column family configuration 2019-02-14 22:35:55,602 DEBUG [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/.tmp/f/131cd200dfb44b5a8bae5fd317fc4129 as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/131cd200dfb44b5a8bae5fd317fc4129 2019-02-14 22:35:55,606 INFO [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.StoreFile(495): HFile Bloom filter type for 131cd200dfb44b5a8bae5fd317fc4129: NONE, but ROW specified in column family configuration 2019-02-14 22:35:55,607 DEBUG [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.HStore(1938): Completing compaction... 2019-02-14 22:35:55,607 INFO [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.HStore(1517): Completed compaction of 1 file(s) in f of TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. into 131cd200dfb44b5a8bae5fd317fc4129(size=4.4 K), total size for store is 4.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2019-02-14 22:35:55,607 DEBUG [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.HRegion(2127): Compaction status journal: Compacting store f at 1550183755175 Compaction complete at 1550183755607 2019-02-14 22:35:55,607 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-0] regionserver.HRegion(1553): Updates disabled for region TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:55,608 INFO [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.CompactSplitThread$CompactionRunner(581): Completed compaction: Request = regionName=TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525., storeName=f, fileCount=1, fileSize=1.3 M (1.3 M), priority=1, time=36216653944224872; duration=0sec 2019-02-14 22:35:55,608 DEBUG [RS:0;ip-172-31-47-35:39948-shortCompactions-1550183747094] regionserver.CompactSplitThread$CompactionRunner(606): CompactSplitThread Status: compaction_queue=(0:0), split_queue=0, merge_queue=0 2019-02-14 22:35:55,608 INFO [RS:0;ip-172-31-47-35:39948] regionserver.CompactSplitThread(445): Waiting for Small Compaction Thread to finish... 2019-02-14 22:35:55,610 INFO [RS:0;ip-172-31-47-35:39948] regionserver.HRegionServer(1359): Waiting on 3 regions to close 2019-02-14 22:35:55,610 DEBUG [RS_CLOSE_META-ip-172-31-47-35:39948-0] handler.CloseRegionHandler(119): Processing close of hbase:meta,,1.1588230740 2019-02-14 22:35:55,610 DEBUG [RS:0;ip-172-31-47-35:39948] regionserver.HRegionServer(1363): {0b19a16b6042d69e3048c196d2a7c525=TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525., 1588230740=hbase:meta,,1.1588230740, c71b20fd2578c13df17ccd07b9ab4dd5=hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5.} 2019-02-14 22:35:55,613 DEBUG [RS_CLOSE_META-ip-172-31-47-35:39948-0] regionserver.HRegion(1526): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2019-02-14 22:35:55,612 DEBUG [StoreCloserThread-TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.-1] regionserver.HStore(2878): Moving the files [hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/9d1822e1d3a447c3a6fdc07c8d3a3c7b, hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/51309c87436c439aa1a005af85f79252, hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/6c664385246141a4bb1aa0301536b8f5, hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/ae7b89c41de44052a04c52f21a98c14c, hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/34bb22bdc7b247b9b10b363517de408e, hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/01700c6968a5474981a43aef16f26aaf, hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/e391bc8d7b314d5e889fbac957ede999, hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/6757a48194e54a27b1fffdb1266c807c, hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/2d2bf9b738554a288d421a80cde645a9, hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/3e3376540ae34b52b351b80a2d1ea6e6] to archive 2019-02-14 22:35:55,613 DEBUG [RS_CLOSE_META-ip-172-31-47-35:39948-0] regionserver.HRegion(1553): Updates disabled for region hbase:meta,,1.1588230740 2019-02-14 22:35:55,613 INFO [RS_CLOSE_META-ip-172-31-47-35:39948-0] regionserver.HRegion(2432): Flushing 1/1 column families, memstore=3.29 KB 2019-02-14 22:35:55,614 DEBUG [StoreCloserThread-TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.-1] backup.HFileArchiver(235): Archiving compacted store files. 2019-02-14 22:35:55,616 DEBUG [RS_CLOSE_META-ip-172-31-47-35:39948-0] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/meta/1588230740/.tmp/info 2019-02-14 22:35:55,616 DEBUG [RS_CLOSE_META-ip-172-31-47-35:39948-0] util.CommonFSUtils(593): Set storagePolicy=HOT for path=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/meta/1588230740/.tmp/info 2019-02-14 22:35:55,616 DEBUG [StoreCloserThread-TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.-1] backup.HFileArchiver(427): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/9d1822e1d3a447c3a6fdc07c8d3a3c7b, to hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/archive/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/9d1822e1d3a447c3a6fdc07c8d3a3c7b 2019-02-14 22:35:55,618 DEBUG [StoreCloserThread-TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.-1] backup.HFileArchiver(427): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/51309c87436c439aa1a005af85f79252, to hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/archive/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/51309c87436c439aa1a005af85f79252 2019-02-14 22:35:55,620 DEBUG [StoreCloserThread-TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.-1] backup.HFileArchiver(427): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/6c664385246141a4bb1aa0301536b8f5, to hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/archive/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/6c664385246141a4bb1aa0301536b8f5 2019-02-14 22:35:55,622 DEBUG [StoreCloserThread-TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.-1] backup.HFileArchiver(427): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/ae7b89c41de44052a04c52f21a98c14c, to hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/archive/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/ae7b89c41de44052a04c52f21a98c14c 2019-02-14 22:35:55,625 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741854_1030{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 7046 2019-02-14 22:35:55,626 DEBUG [StoreCloserThread-TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.-1] backup.HFileArchiver(427): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/34bb22bdc7b247b9b10b363517de408e, to hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/archive/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/34bb22bdc7b247b9b10b363517de408e 2019-02-14 22:35:55,629 DEBUG [StoreCloserThread-TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.-1] backup.HFileArchiver(427): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/01700c6968a5474981a43aef16f26aaf, to hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/archive/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/01700c6968a5474981a43aef16f26aaf 2019-02-14 22:35:55,630 DEBUG [StoreCloserThread-TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.-1] backup.HFileArchiver(427): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/e391bc8d7b314d5e889fbac957ede999, to hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/archive/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/e391bc8d7b314d5e889fbac957ede999 2019-02-14 22:35:55,632 DEBUG [StoreCloserThread-TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.-1] backup.HFileArchiver(427): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/6757a48194e54a27b1fffdb1266c807c, to hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/archive/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/6757a48194e54a27b1fffdb1266c807c 2019-02-14 22:35:55,633 DEBUG [StoreCloserThread-TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.-1] backup.HFileArchiver(427): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/2d2bf9b738554a288d421a80cde645a9, to hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/archive/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/2d2bf9b738554a288d421a80cde645a9 2019-02-14 22:35:55,633 INFO [ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583_ChoreService_1] hbase.ScheduledChore(183): Chore: ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583-DoMetricsChore was stopped 2019-02-14 22:35:55,635 DEBUG [StoreCloserThread-TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.-1] backup.HFileArchiver(427): Finished archiving from class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile, file:hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/3e3376540ae34b52b351b80a2d1ea6e6, to hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/archive/data/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/f/3e3376540ae34b52b351b80a2d1ea6e6 2019-02-14 22:35:55,636 INFO [StoreCloserThread-TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525.-1] regionserver.HStore(954): Closed f 2019-02-14 22:35:55,639 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-0] wal.WALSplitter(751): Wrote region seqId=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/default/TestFIFOCompactionPolicy/0b19a16b6042d69e3048c196d2a7c525/recovered.edits/146.seqid to file, newSeqId=146, maxSeqId=2 2019-02-14 22:35:55,641 INFO [RS_CLOSE_REGION-ip-172-31-47-35:39948-0] regionserver.HRegion(1660): Closed TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:55,641 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-0] handler.CloseRegionHandler(164): Closed TestFIFOCompactionPolicy,,1550183747397.0b19a16b6042d69e3048c196d2a7c525. 2019-02-14 22:35:55,668 INFO [M:0;ip-172-31-47-35:40925] regionserver.HRegionServer(1124): stopping server ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583 2019-02-14 22:35:55,668 DEBUG [M:0;ip-172-31-47-35:40925] zookeeper.MetaTableLocator(623): Stopping MetaTableLocator 2019-02-14 22:35:55,669 INFO [M:0;ip-172-31-47-35:40925] client.ConnectionManager$HConnectionImplementation(1774): Closing zookeeper sessionid=0x168ee26a5f90002 2019-02-14 22:35:55,671 DEBUG [M:0;ip-172-31-47-35:40925] ipc.AbstractRpcClient(483): Stopping rpc client 2019-02-14 22:35:55,671 DEBUG [RpcServer.reader=2,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=39948] ipc.RpcServer$Listener(971): RpcServer.listener,port=39948: DISCONNECTING client 172.31.47.35:51716 because read count=-1. Number of active connections: 3 2019-02-14 22:35:55,671 INFO [M:0;ip-172-31-47-35:40925] regionserver.HRegionServer(1152): stopping server ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583; all regions closed. 2019-02-14 22:35:55,672 INFO [M:0;ip-172-31-47-35:40925] hbase.ChoreService(328): Chore service for: ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583 had [[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS]] on shutdown 2019-02-14 22:35:55,672 INFO [M:0;ip-172-31-47-35:40925] master.ServerManager(574): Waiting on regionserver(s) to go down ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:55,681 INFO [RS_CLOSE_REGION-ip-172-31-47-35:39948-1] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=7, memsize=248, hasBloomFilter=true, into tmp file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/namespace/c71b20fd2578c13df17ccd07b9ab4dd5/.tmp/info/b5d9b646c82d420095a614fe38b6681c 2019-02-14 22:35:55,687 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-1] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/namespace/c71b20fd2578c13df17ccd07b9ab4dd5/.tmp/info/b5d9b646c82d420095a614fe38b6681c as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/namespace/c71b20fd2578c13df17ccd07b9ab4dd5/info/b5d9b646c82d420095a614fe38b6681c 2019-02-14 22:35:55,693 INFO [RS_CLOSE_REGION-ip-172-31-47-35:39948-1] regionserver.HStore(1050): Added hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/namespace/c71b20fd2578c13df17ccd07b9ab4dd5/info/b5d9b646c82d420095a614fe38b6681c, entries=2, sequenceid=7, filesize=4.8 K 2019-02-14 22:35:55,694 INFO [RS_CLOSE_REGION-ip-172-31-47-35:39948-1] regionserver.HRegion(2732): Finished memstore flush of ~248 B/248, currentsize=0 B/0 for region hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5. in 432ms, sequenceid=7, compaction requested=false 2019-02-14 22:35:55,695 INFO [StoreCloserThread-hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5.-1] regionserver.HStore(954): Closed info 2019-02-14 22:35:55,699 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-1] wal.WALSplitter(751): Wrote region seqId=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/hbase/namespace/c71b20fd2578c13df17ccd07b9ab4dd5/recovered.edits/10.seqid to file, newSeqId=10, maxSeqId=2 2019-02-14 22:35:55,700 INFO [RS_CLOSE_REGION-ip-172-31-47-35:39948-1] regionserver.HRegion(1660): Closed hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5. 2019-02-14 22:35:55,700 DEBUG [RS_CLOSE_REGION-ip-172-31-47-35:39948-1] handler.CloseRegionHandler(164): Closed hbase:namespace,,1550183740592.c71b20fd2578c13df17ccd07b9ab4dd5. 2019-02-14 22:35:56,027 INFO [RS_CLOSE_META-ip-172-31-47-35:39948-0] regionserver.DefaultStoreFlusher(91): Flushed, sequenceid=12, memsize=3.3 K, hasBloomFilter=false, into tmp file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/meta/1588230740/.tmp/info/21cc4049197f44a29f63faa729a9a6df 2019-02-14 22:35:56,033 DEBUG [RS_CLOSE_META-ip-172-31-47-35:39948-0] regionserver.HRegionFileSystem(460): Committing store file hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/meta/1588230740/.tmp/info/21cc4049197f44a29f63faa729a9a6df as hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/meta/1588230740/info/21cc4049197f44a29f63faa729a9a6df 2019-02-14 22:35:56,039 INFO [RS_CLOSE_META-ip-172-31-47-35:39948-0] regionserver.HStore(1050): Added hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/data/hbase/meta/1588230740/info/21cc4049197f44a29f63faa729a9a6df, entries=15, sequenceid=12, filesize=6.9 K 2019-02-14 22:35:56,041 INFO [RS_CLOSE_META-ip-172-31-47-35:39948-0] regionserver.HRegion(2732): Finished memstore flush of ~3.29 KB/3368, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 427ms, sequenceid=12, compaction requested=false 2019-02-14 22:35:56,042 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(954): Closed info 2019-02-14 22:35:56,046 DEBUG [RS_CLOSE_META-ip-172-31-47-35:39948-0] wal.WALSplitter(751): Wrote region seqId=hdfs://localhost:41444/user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/hbase/meta/1588230740/recovered.edits/15.seqid to file, newSeqId=15, maxSeqId=3 2019-02-14 22:35:56,047 DEBUG [RS_CLOSE_META-ip-172-31-47-35:39948-0] coprocessor.CoprocessorHost(302): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2019-02-14 22:35:56,047 INFO [RS_CLOSE_META-ip-172-31-47-35:39948-0] regionserver.HRegion(1660): Closed hbase:meta,,1.1588230740 2019-02-14 22:35:56,047 DEBUG [RS_CLOSE_META-ip-172-31-47-35:39948-0] handler.CloseRegionHandler(164): Closed hbase:meta,,1.1588230740 2019-02-14 22:35:56,101 INFO [ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583_splitLogManager__ChoreService_1] hbase.ScheduledChore(183): Chore: SplitLogManager Timeout Monitor was stopped 2019-02-14 22:35:56,213 INFO [RS:0;ip-172-31-47-35:39948] regionserver.HRegionServer(1152): stopping server ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985; all regions closed. 2019-02-14 22:35:56,213 DEBUG [RS:0;ip-172-31-47-35:39948] wal.FSHLog(1105): Closing WAL writer in /user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/WALs/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:56,216 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741829_1005{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f3ca40dc-91f2-42ec-82fe-8f7a3a14736f:NORMAL:127.0.0.1:38317|RBW]]} size 3302 2019-02-14 22:35:56,227 INFO [RS_OPEN_META-ip-172-31-47-35:39948-0-MetaLogRoller] regionserver.LogRoller(201): LogRoller exiting. 2019-02-14 22:35:56,618 DEBUG [RS:0;ip-172-31-47-35:39948] wal.FSHLog(1063): Moved 1 WAL file(s) to /user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/oldWALs 2019-02-14 22:35:56,618 INFO [RS:0;ip-172-31-47-35:39948] wal.FSHLog(1066): Closed WAL: FSHLog ip-172-31-47-35.us-west-2.compute.internal%2C39948%2C1550183737985.meta:.meta(num 1550183740240) 2019-02-14 22:35:56,618 DEBUG [RS:0;ip-172-31-47-35:39948] wal.FSHLog(1105): Closing WAL writer in /user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/WALs/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:56,621 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38317 is added to blk_1073741830_1006{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-4c27f4c9-2d8c-41fa-8b13-bd794c8bac6c:NORMAL:127.0.0.1:38317|RBW]]} size 13129997 2019-02-14 22:35:56,676 INFO [M:0;ip-172-31-47-35:40925] master.ServerManager(574): Waiting on regionserver(s) to go down ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:57,023 DEBUG [RS:0;ip-172-31-47-35:39948] wal.FSHLog(1063): Moved 1 WAL file(s) to /user/apurtell/test-data/e1d6567d-e7e4-4fbb-80db-768dde36d862/oldWALs 2019-02-14 22:35:57,023 INFO [RS:0;ip-172-31-47-35:39948] wal.FSHLog(1066): Closed WAL: FSHLog ip-172-31-47-35.us-west-2.compute.internal%2C39948%2C1550183737985:(num 1550183740526) 2019-02-14 22:35:57,023 DEBUG [RS:0;ip-172-31-47-35:39948] ipc.AbstractRpcClient(483): Stopping rpc client 2019-02-14 22:35:57,023 DEBUG [RpcServer.reader=1,bindAddress=ip-172-31-47-35.us-west-2.compute.internal,port=40925] ipc.RpcServer$Listener(971): RpcServer.listener,port=40925: DISCONNECTING client 172.31.47.35:55682 because read count=-1. Number of active connections: 1 2019-02-14 22:35:57,023 INFO [RS:0;ip-172-31-47-35:39948] regionserver.Leases(147): RS:0;ip-172-31-47-35:39948 closing leases 2019-02-14 22:35:57,024 INFO [RS:0;ip-172-31-47-35:39948] regionserver.Leases(150): RS:0;ip-172-31-47-35:39948 closed leases 2019-02-14 22:35:57,024 INFO [RS:0;ip-172-31-47-35:39948] hbase.ChoreService(328): Chore service for: ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 had [[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS]] on shutdown 2019-02-14 22:35:57,028 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:57,028 INFO [RS:0;ip-172-31-47-35:39948] ipc.RpcServer(2493): Stopping server on 39948 2019-02-14 22:35:57,028 INFO [RpcServer.listener,port=39948] ipc.RpcServer$Listener(875): RpcServer.listener,port=39948: stopping 2019-02-14 22:35:57,029 INFO [RpcServer.responder] ipc.RpcServer$Responder(1118): RpcServer.responder: stopped 2019-02-14 22:35:57,029 INFO [RpcServer.responder] ipc.RpcServer$Responder(1021): RpcServer.responder: stopping 2019-02-14 22:35:57,032 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:57,032 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 2019-02-14 22:35:57,032 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): regionserver:39948-0x168ee26a5f90001, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2019-02-14 22:35:57,034 INFO [main-EventThread] zookeeper.RegionServerTracker(125): RegionServer ephemeral node deleted, processing expiration [ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985] 2019-02-14 22:35:57,036 DEBUG [main-EventThread] master.DeadServer(133): Added ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985; numProcessing=1 2019-02-14 22:35:57,036 INFO [main-EventThread] master.ServerManager(634): Cluster shutdown set; ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985 expired; onlineServers=0 2019-02-14 22:35:57,036 DEBUG [M:0;ip-172-31-47-35:40925] master.HMaster(1300): Stopping service threads 2019-02-14 22:35:57,036 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2019-02-14 22:35:57,037 INFO [RS:0;ip-172-31-47-35:39948] regionserver.HRegionServer(1196): stopping server ip-172-31-47-35.us-west-2.compute.internal,39948,1550183737985; zookeeper connection closed. 2019-02-14 22:35:57,037 INFO [RS:0;ip-172-31-47-35:39948] regionserver.HRegionServer(1199): RS:0;ip-172-31-47-35:39948 exiting 2019-02-14 22:35:57,037 DEBUG [M:0;ip-172-31-47-35:40925] zookeeper.ZKUtil(633): master:40925-0x168ee26a5f90000, quorum=localhost:50587, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2019-02-14 22:35:57,037 WARN [M:0;ip-172-31-47-35:40925] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2019-02-14 22:35:57,037 INFO [M:0;ip-172-31-47-35:40925] client.ConnectionManager$HConnectionImplementation(1774): Closing zookeeper sessionid=0x168ee26a5f90004 2019-02-14 22:35:57,037 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@35add18d] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(195): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@35add18d 2019-02-14 22:35:57,038 INFO [main] util.JVMClusterUtil(331): Shutdown of 1 master(s) and 1 regionserver(s) complete 2019-02-14 22:35:57,039 DEBUG [M:0;ip-172-31-47-35:40925] ipc.AbstractRpcClient(483): Stopping rpc client 2019-02-14 22:35:57,039 INFO [M:0;ip-172-31-47-35:40925] hbase.ChoreService(328): Chore service for: ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583_splitLogManager_ had [] on shutdown 2019-02-14 22:35:57,039 INFO [M:0;ip-172-31-47-35:40925] flush.MasterFlushTableProcedureManager(79): stop: server shutting down. 2019-02-14 22:35:57,039 INFO [M:0;ip-172-31-47-35:40925] ipc.RpcServer(2493): Stopping server on 40925 2019-02-14 22:35:57,039 INFO [RpcServer.listener,port=40925] ipc.RpcServer$Listener(875): RpcServer.listener,port=40925: stopping 2019-02-14 22:35:57,041 INFO [RpcServer.responder] ipc.RpcServer$Responder(1118): RpcServer.responder: stopped 2019-02-14 22:35:57,041 INFO [RpcServer.responder] ipc.RpcServer$Responder(1021): RpcServer.responder: stopping 2019-02-14 22:35:57,043 DEBUG [M:0;ip-172-31-47-35:40925] zookeeper.RecoverableZooKeeper(188): Node /hbase/rs/ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583 already deleted, retry=false 2019-02-14 22:35:57,047 INFO [M:0;ip-172-31-47-35:40925] regionserver.HRegionServer(1196): stopping server ip-172-31-47-35.us-west-2.compute.internal,40925,1550183737583; zookeeper connection closed. 2019-02-14 22:35:57,047 INFO [M:0;ip-172-31-47-35:40925] regionserver.HRegionServer(1199): M:0;ip-172-31-47-35:40925 exiting 2019-02-14 22:35:57,047 ERROR [main] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2019-02-14 22:35:57,048 INFO [main] zookeeper.MiniZooKeeperCluster(326): Shutdown MiniZK cluster with all ZK servers 2019-02-14 22:35:57,048 WARN [main] datanode.DirectoryScanner(529): DirectoryScanner: shutdown has been called 2019-02-14 22:35:57,062 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2019-02-14 22:35:57,147 DEBUG [ip-172-31-47-35:40925.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(615): replicationLogCleaner-0x168ee26a5f90005, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2019-02-14 22:35:57,147 DEBUG [ip-172-31-47-35:40925.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(683): replicationLogCleaner-0x168ee26a5f90005, quorum=localhost:50587, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2019-02-14 22:35:57,147 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(615): hconnection-0x1cb7936c-0x168ee26a5f90009, quorum=localhost:50587, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2019-02-14 22:35:57,148 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(683): hconnection-0x1cb7936c-0x168ee26a5f90009, quorum=localhost:50587, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2019-02-14 22:35:57,164 WARN [DataNode: [[[DISK]file:/data/src/hbase/hbase-server/target/test-data/5a4d94bc-56f3-4901-bee1-e276bd3b8f91/dfscluster_af8fd26e-e004-4f4b-9256-660ce931a171/dfs/data/data1/, [DISK]file:/data/src/hbase/hbase-server/target/test-data/5a4d94bc-56f3-4901-bee1-e276bd3b8f91/dfscluster_af8fd26e-e004-4f4b-9256-660ce931a171/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:41444] datanode.IncrementalBlockReportManager(132): IncrementalBlockReportManager interrupted 2019-02-14 22:35:57,164 WARN [DataNode: [[[DISK]file:/data/src/hbase/hbase-server/target/test-data/5a4d94bc-56f3-4901-bee1-e276bd3b8f91/dfscluster_af8fd26e-e004-4f4b-9256-660ce931a171/dfs/data/data1/, [DISK]file:/data/src/hbase/hbase-server/target/test-data/5a4d94bc-56f3-4901-bee1-e276bd3b8f91/dfscluster_af8fd26e-e004-4f4b-9256-660ce931a171/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:41444] datanode.BPServiceActor(670): Ending block pool service for: Block pool BP-1808774508-172.31.47.35-1550183734971 (Datanode Uuid b9a2f5fe-b4d7-4d90-97aa-c764a9863eae) service to localhost/127.0.0.1:41444 2019-02-14 22:35:57,186 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2019-02-14 22:35:57,305 INFO [main] hbase.HBaseTestingUtility(1185): Minicluster is down 2019-02-14 22:35:57,346 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@ba2f4ec 2019-02-14 22:35:57,347 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(120): Starting fs shutdown hook thread. 2019-02-14 22:35:57,347 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(134): Shutdown hook finished.