Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15646 Track failing tests in HDFS
  3. HDFS-15654

TestBPOfferService#testMissBlocksWhenReregister fails intermittently

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      TestBPOfferService.testMissBlocksWhenReregister is flaky. It fails randomly when the
      following expression is not true:

            assertTrue(fullBlockReportCount == totalTestBlocks ||
                incrBlockReportCount == totalTestBlocks);
      

      There is a race condition here that relies once more on "time" to synchronize between concurrent threads. The code below is is causing the non-deterministic execution.
      On a slow server, addNewBlockThread may not be done by the time the main thread reach the assertion call.

            // Verify FBR/IBR count is equal to generate number.
            assertTrue(fullBlockReportCount == totalTestBlocks ||
                incrBlockReportCount == totalTestBlocks);
          } finally {
            addNewBlockThread.join();
            bpos.stop();
            bpos.join();
      

      Therefore, the correct implementation should wait for the thread to finish

           // the thread finished execution.
           addNewBlockThread.join();
            // Verify FBR/IBR count is equal to generate number.
            assertTrue(fullBlockReportCount == totalTestBlocks ||
                incrBlockReportCount == totalTestBlocks);
          } finally {
            bpos.stop();
            bpos.join();
      

      DataNodeFaultInjector needs to have a longer wait_time too. 1 second is not enough to satisfy the condition.

            DataNodeFaultInjector.set(new DataNodeFaultInjector() {
              public void blockUtilSendFullBlockReport() {
                try {
                  GenericTestUtils.waitFor(() -> {
                    if(count.get() > 2000) {
                      return true;
                    }
                    return false;
                  }, 100, 10000); // increase that waiting time to 10 seconds.
                } catch (Exception e) {
                  e.printStackTrace();
                }
              }
            });
      
      Stacktrace
      java.lang.AssertionError
      	at org.junit.Assert.fail(Assert.java:86)
      	at org.junit.Assert.assertTrue(Assert.java:41)
      	at org.junit.Assert.assertTrue(Assert.java:52)
      	at org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testMissBlocksWhenReregister(TestBPOfferService.java:350)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
      	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
      	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
      	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
      	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
      	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
      Standard Output
      2020-10-26 07:38:56,442 [main] INFO  common.Util (Util.java:isDiskStatsEnabled(395)) - dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
      2020-10-26 07:38:56,454 [main] INFO  datanode.DataNode (SimulatedFSDataset.java:registerMBean(1370)) - Registered FSDatasetState MBean
      2020-10-26 07:38:56,455 [main] INFO  common.Util (Util.java:isDiskStatsEnabled(395)) - dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
      2020-10-26 07:38:56,460 [Thread-103] INFO  datanode.DataNode (BPServiceActor.java:run(846)) - Block pool <registering> (Datanode Uuid unassigned) service to 0.0.0.0/0.0.0.0:0 starting to offer service
      2020-10-26 07:38:56,461 [Thread-104] INFO  datanode.DataNode (BPServiceActor.java:run(846)) - Block pool <registering> (Datanode Uuid unassigned) service to 0.0.0.0/0.0.0.0:1 starting to offer service
      2020-10-26 07:38:56,463 [Thread-103] DEBUG datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(245)) - Block pool <registering> (Datanode Uuid unassigned) service to 0.0.0.0/0.0.0.0:0 received versionRequest response: lv=-65;cid=fake cluster;nsid=1;c=0;bpid=fake bpid
      2020-10-26 07:38:56,464 [Thread-104] DEBUG datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(245)) - Block pool <registering> (Datanode Uuid unassigned) service to 0.0.0.0/0.0.0.0:1 received versionRequest response: lv=-65;cid=fake cluster;nsid=1;c=0;bpid=fake bpid
      2020-10-26 07:38:56,466 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] INFO  datanode.DataNode (BPServiceActor.java:register(787)) - Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0 beginning handshake with NN
      2020-10-26 07:38:56,466 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] INFO  datanode.DataNode (BPServiceActor.java:register(787)) - Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:1 beginning handshake with NN
      2020-10-26 07:38:56,467 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] INFO  datanode.DataNode (BPServiceActor.java:register(815)) - Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0 successfully registered with NN
      2020-10-26 07:38:56,467 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] INFO  datanode.DataNode (BPServiceActor.java:offerService(644)) - For namenode 0.0.0.0/0.0.0.0:0 using BLOCKREPORT_INTERVAL of 21600000msecs CACHEREPORT_INTERVAL of 10000msecs Initial delay: 0msecs; heartBeatInterval=3000
      2020-10-26 07:38:56,467 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] INFO  datanode.DataNode (BPServiceActor.java:register(815)) - Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:1 successfully registered with NN
      2020-10-26 07:38:56,469 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] INFO  datanode.DataNode (BPServiceActor.java:offerService(644)) - For namenode 0.0.0.0/0.0.0.0:1 using BLOCKREPORT_INTERVAL of 21600000msecs CACHEREPORT_INTERVAL of 10000msecs Initial delay: 0msecs; heartBeatInterval=3000
      2020-10-26 07:38:56,469 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] DEBUG datanode.DataNode (BPServiceActor.java:sendHeartBeat(522)) - Sending heartbeat with 1 storage reports from service actor: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0
      2020-10-26 07:38:56,470 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] DEBUG datanode.DataNode (BPServiceActor.java:sendHeartBeat(522)) - Sending heartbeat with 1 storage reports from service actor: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:1
      2020-10-26 07:38:56,471 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] INFO  datanode.TestBPOfferService (TestBPOfferService.java:answer(205)) - fullBlockReportLeaseId=1
      2020-10-26 07:38:56,471 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] INFO  datanode.TestBPOfferService (TestBPOfferService.java:answer(205)) - fullBlockReportLeaseId=2
      2020-10-26 07:38:56,474 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] INFO  datanode.DataNode (BPServiceActor.java:blockReport(437)) - Successfully sent block report 0x7688fa248c41cdeb,  containing 1 storage report(s), of which we sent 1. The reports had 0 total blocks and used 1 RPC(s). This took 1 msecs to generate and 0 msecs for RPC and NN processing. Got back no commands.
      2020-10-26 07:38:56,475 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] INFO  datanode.DataNode (BPServiceActor.java:blockReport(437)) - Successfully sent block report 0x7fd3be5ab5d82bff,  containing 1 storage report(s), of which we sent 1. The reports had 0 total blocks and used 1 RPC(s). This took 1 msecs to generate and 0 msecs for RPC and NN processing. Got back no commands.
      2020-10-26 07:38:56,669 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] DEBUG datanode.DataNode (BPServiceActor.java:sendHeartBeat(522)) - Sending heartbeat with 1 storage reports from service actor: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0
      2020-10-26 07:38:56,670 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] INFO  datanode.TestBPOfferService (TestBPOfferService.java:answer(205)) - fullBlockReportLeaseId=0
      2020-10-26 07:38:56,671 [Command processor] INFO  datanode.DataNode (BPOfferService.java:processCommandFromActor(674)) - DatanodeCommand action : DNA_REGISTER from 0.0.0.0/0.0.0.0:0 with standby state
      2020-10-26 07:38:56,673 [Command processor] DEBUG datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(245)) - Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0 received versionRequest response: lv=-65;cid=fake cluster;nsid=1;c=0;bpid=fake bpid
      2020-10-26 07:38:56,674 [Command processor] INFO  datanode.DataNode (BPServiceActor.java:register(787)) - Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0 beginning handshake with NN
      2020-10-26 07:38:56,675 [Command processor] INFO  datanode.DataNode (BPServiceActor.java:register(815)) - Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0 successfully registered with NN
      2020-10-26 07:38:56,770 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] DEBUG datanode.DataNode (BPServiceActor.java:sendHeartBeat(522)) - Sending heartbeat with 1 storage reports from service actor: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:1
      2020-10-26 07:38:56,770 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] INFO  datanode.TestBPOfferService (TestBPOfferService.java:answer(205)) - fullBlockReportLeaseId=0
      2020-10-26 07:38:59,669 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] DEBUG datanode.DataNode (BPServiceActor.java:sendHeartBeat(522)) - Sending heartbeat with 1 storage reports from service actor: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0
      2020-10-26 07:38:59,670 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] INFO  datanode.TestBPOfferService (TestBPOfferService.java:answer(205)) - fullBlockReportLeaseId=3
      2020-10-26 07:38:59,681 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] INFO  datanode.DataNode (BPServiceActor.java:blockReport(437)) - Successfully sent block report 0x7688fa248c41cdec,  containing 1 storage report(s), of which we sent 1. The reports had 1782 total blocks and used 1 RPC(s). This took 9 msecs to generate and 1 msecs for RPC and NN processing. Got back no commands.
      2020-10-26 07:38:59,770 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] DEBUG datanode.DataNode (BPServiceActor.java:sendHeartBeat(522)) - Sending heartbeat with 1 storage reports from service actor: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:1
      2020-10-26 07:38:59,771 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] INFO  datanode.TestBPOfferService (TestBPOfferService.java:answer(205)) - fullBlockReportLeaseId=0
      2020-10-26 07:39:02,669 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] DEBUG datanode.DataNode (BPServiceActor.java:sendHeartBeat(522)) - Sending heartbeat with 1 storage reports from service actor: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0
      2020-10-26 07:39:02,669 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] INFO  datanode.TestBPOfferService (TestBPOfferService.java:answer(205)) - fullBlockReportLeaseId=0
      2020-10-26 07:39:02,770 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] DEBUG datanode.DataNode (BPServiceActor.java:sendHeartBeat(522)) - Sending heartbeat with 1 storage reports from service actor: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:1
      2020-10-26 07:39:02,771 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] INFO  datanode.TestBPOfferService (TestBPOfferService.java:answer(205)) - fullBlockReportLeaseId=0
      2020-10-26 07:39:05,669 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] DEBUG datanode.DataNode (BPServiceActor.java:sendHeartBeat(522)) - Sending heartbeat with 1 storage reports from service actor: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0
      2020-10-26 07:39:05,669 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] INFO  datanode.TestBPOfferService (TestBPOfferService.java:answer(205)) - fullBlockReportLeaseId=0
      2020-10-26 07:39:05,770 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] DEBUG datanode.DataNode (BPServiceActor.java:sendHeartBeat(522)) - Sending heartbeat with 1 storage reports from service actor: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:1
      2020-10-26 07:39:05,771 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] INFO  datanode.TestBPOfferService (TestBPOfferService.java:answer(205)) - fullBlockReportLeaseId=0
      2020-10-26 07:39:08,669 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] DEBUG datanode.DataNode (BPServiceActor.java:sendHeartBeat(522)) - Sending heartbeat with 1 storage reports from service actor: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0
      2020-10-26 07:39:08,670 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] INFO  datanode.TestBPOfferService (TestBPOfferService.java:answer(205)) - fullBlockReportLeaseId=0
      2020-10-26 07:39:08,771 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] DEBUG datanode.DataNode (BPServiceActor.java:sendHeartBeat(522)) - Sending heartbeat with 1 storage reports from service actor: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:1
      2020-10-26 07:39:08,771 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] INFO  datanode.TestBPOfferService (TestBPOfferService.java:answer(205)) - fullBlockReportLeaseId=0
      2020-10-26 07:39:11,669 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] DEBUG datanode.DataNode (BPServiceActor.java:sendHeartBeat(522)) - Sending heartbeat with 1 storage reports from service actor: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0
      2020-10-26 07:39:11,669 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] INFO  datanode.TestBPOfferService (TestBPOfferService.java:answer(205)) - fullBlockReportLeaseId=0
      2020-10-26 07:39:11,771 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] DEBUG datanode.DataNode (BPServiceActor.java:sendHeartBeat(522)) - Sending heartbeat with 1 storage reports from service actor: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:1
      2020-10-26 07:39:11,772 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] INFO  datanode.TestBPOfferService (TestBPOfferService.java:answer(205)) - fullBlockReportLeaseId=0
      2020-10-26 07:39:11,878 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] WARN  datanode.IncrementalBlockReportManager (IncrementalBlockReportManager.java:waitTillNextIBR(160)) - IncrementalBlockReportManager interrupted
      2020-10-26 07:39:11,878 [Command processor] ERROR datanode.DataNode (BPServiceActor.java:processQueue(1329)) - Command processor encountered interrupt and exit.
      2020-10-26 07:39:11,878 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] WARN  datanode.IncrementalBlockReportManager (IncrementalBlockReportManager.java:waitTillNextIBR(160)) - IncrementalBlockReportManager interrupted
      2020-10-26 07:39:11,878 [Command processor] ERROR datanode.DataNode (BPServiceActor.java:processQueue(1329)) - Command processor encountered interrupt and exit.
      2020-10-26 07:39:11,879 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:1] WARN  datanode.DataNode (BPServiceActor.java:run(889)) - Ending block pool service for: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:1
      2020-10-26 07:39:11,878 [fake bpid heartbeating to 0.0.0.0/0.0.0.0:0] WARN  datanode.DataNode (BPServiceActor.java:run(889)) - Ending block pool service for: Block pool fake bpid (Datanode Uuid null) service to 0.0.0.0/0.0.0.0:0
      Standard Error
      java.util.concurrent.TimeoutException: Timed out waiting for condition. Thread diagnostics:
      Timestamp: 2020-10-26 07:38:57,678
      
      "nioEventLoopGroup-2-30"  prio=10 tid=190 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "fake bpid heartbeating to 0.0.0.0/0.0.0.0:1" daemon prio=5 tid=260 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Object.wait(Native Method)
              at org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:158)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:738)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:878)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-27"  prio=10 tid=187 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "Signal Dispatcher" daemon prio=9 tid=4 runnable
      java.lang.Thread.State: RUNNABLE
      "nioEventLoopGroup-2-4"  prio=10 tid=164 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-10"  prio=10 tid=170 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-41"  prio=10 tid=201 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-19"  prio=10 tid=179 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-1"  prio=10 tid=108 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-40"  prio=10 tid=200 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-36"  prio=10 tid=244 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-20"  prio=10 tid=180 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-6"  prio=10 tid=214 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "process reaper" daemon prio=10 tid=20 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
              at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      "Command processor" daemon prio=5 tid=258 in Object.wait()
      java.lang.Thread.State: WAITING (on object monitor)
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processQueue(BPServiceActor.java:1324)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.run(BPServiceActor.java:1312)
      "nioEventLoopGroup-2-21"  prio=10 tid=181 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-7"  prio=10 tid=215 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-5"  prio=10 tid=165 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-8"  prio=10 tid=168 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "GcTimeMonitor obsWindow = 60000, sleepInterval = 5000, maxGcTimePerc = 100" daemon prio=5 tid=42 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at org.apache.hadoop.util.GcTimeMonitor.run(GcTimeMonitor.java:155)
      "nioEventLoopGroup-3-5"  prio=10 tid=213 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "Finalizer" daemon prio=8 tid=3 in Object.wait()
      java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
              at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
      "nioEventLoopGroup-3-32"  prio=10 tid=240 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "surefire-forkedjvm-ping-30s" daemon prio=5 tid=19 runnable
      java.lang.Thread.State: RUNNABLE
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-3"  prio=10 tid=163 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-47"  prio=10 tid=207 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-2"  prio=10 tid=210 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-43"  prio=10 tid=251 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-21"  prio=10 tid=229 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-46"  prio=10 tid=206 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-34"  prio=10 tid=194 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-37"  prio=10 tid=197 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-33"  prio=10 tid=241 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-4"  prio=10 tid=212 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-38"  prio=10 tid=246 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-25"  prio=10 tid=233 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-39"  prio=10 tid=247 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-46"  prio=10 tid=254 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "main"  prio=5 tid=1 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:394)
              at org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testMissBlocksWhenReregister(TestBPOfferService.java:340)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
              at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
              at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
              at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
              at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
              at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
              at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
              at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
              at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
              at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
              at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
              at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
              at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
              at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
              at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
              at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
              at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
              at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
              at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
              at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
              at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
              at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
      "nioEventLoopGroup-2-43"  prio=10 tid=203 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-30"  prio=10 tid=238 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-28"  prio=10 tid=236 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-48"  prio=10 tid=256 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-8"  prio=10 tid=216 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "process reaper" daemon prio=10 tid=140 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
              at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
              at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-42"  prio=10 tid=250 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-19"  prio=10 tid=227 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-38"  prio=10 tid=198 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "ForkJoinPool-2-worker-11" daemon prio=5 tid=148 in Object.wait()
      java.lang.Thread.State: WAITING (on object monitor)
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
              at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
              at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
      "nioEventLoopGroup-2-14"  prio=10 tid=174 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-23"  prio=10 tid=183 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-16"  prio=10 tid=176 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-13"  prio=10 tid=221 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-9"  prio=10 tid=217 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-48"  prio=10 tid=208 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-3"  prio=10 tid=211 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "Reference Handler" daemon prio=10 tid=2 in Object.wait()
      java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
              at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
      "nioEventLoopGroup-3-16"  prio=10 tid=224 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-28"  prio=10 tid=188 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-14"  prio=10 tid=222 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "Command processor" daemon prio=5 tid=257 runnable
      java.lang.Thread.State: RUNNABLE
              at java.lang.Thread.dumpThreads(Native Method)
              at java.lang.Thread.getAllStackTraces(Thread.java:1610)
              at org.apache.hadoop.test.TimedOutTestsListener.buildThreadDump(TimedOutTestsListener.java:87)
              at org.apache.hadoop.test.TimedOutTestsListener.buildThreadDiagnosticString(TimedOutTestsListener.java:73)
              at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:401)
              at org.apache.hadoop.hdfs.server.datanode.TestBPOfferService$1.blockUtilSendFullBlockReport(TestBPOfferService.java:295)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.reRegister(BPServiceActor.java:927)
              at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:676)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processCommand(BPServiceActor.java:1351)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.lambda$enqueue$2(BPServiceActor.java:1397)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread$$Lambda$14/1065143224.run(Unknown Source)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processQueue(BPServiceActor.java:1325)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.run(BPServiceActor.java:1312)
      "nioEventLoopGroup-3-15"  prio=10 tid=223 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-40"  prio=10 tid=248 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-31"  prio=10 tid=191 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-25"  prio=10 tid=185 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-37"  prio=10 tid=245 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-45"  prio=10 tid=205 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-24"  prio=10 tid=232 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "ForkJoinPool-2-worker-22" daemon prio=5 tid=151 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
              at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
              at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
      "nioEventLoopGroup-3-45"  prio=10 tid=253 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-35"  prio=10 tid=243 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-22"  prio=10 tid=230 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "LeaseRenewer:jenkins@localhost:35601" daemon prio=5 tid=158 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:412)
              at org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76)
              at org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:308)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-10"  prio=10 tid=218 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "ForkJoinPool-2-worker-18" daemon prio=5 tid=147 in Object.wait()
      java.lang.Thread.State: WAITING (on object monitor)
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
              at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
              at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
      "nioEventLoopGroup-2-12"  prio=10 tid=172 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-2"  prio=10 tid=162 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-44"  prio=10 tid=252 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-29"  prio=10 tid=237 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "client DomainSocketWatcher" daemon prio=5 tid=132 runnable
      java.lang.Thread.State: RUNNABLE
              at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
              at org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(DomainSocketWatcher.java:52)
              at org.apache.hadoop.net.unix.DomainSocketWatcher$2.run(DomainSocketWatcher.java:503)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-44"  prio=10 tid=204 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-22"  prio=10 tid=182 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-36"  prio=10 tid=196 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-33"  prio=10 tid=193 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-6"  prio=10 tid=166 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-24"  prio=10 tid=184 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-23"  prio=10 tid=231 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-18"  prio=10 tid=178 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-15"  prio=10 tid=175 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-35"  prio=10 tid=195 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "Thread-105"  prio=5 tid=261 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.lambda$testMissBlocksWhenReregister$0(TestBPOfferService.java:319)
              at org.apache.hadoop.hdfs.server.datanode.TestBPOfferService$$Lambda$80/374646930.run(Unknown Source)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-12"  prio=10 tid=220 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" daemon prio=5 tid=95 in Object.wait()
      java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
              at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3937)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-42"  prio=10 tid=202 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-31"  prio=10 tid=239 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-20"  prio=10 tid=228 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-17"  prio=10 tid=225 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-9"  prio=10 tid=169 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-32"  prio=10 tid=192 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "AsyncAppender-Dispatcher-Thread-72" daemon prio=5 tid=118 in Object.wait()
      java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.log4j.AsyncAppender$Dispatcher.run(AsyncAppender.java:548)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-17"  prio=10 tid=177 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-18"  prio=10 tid=226 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-13"  prio=10 tid=173 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "fake bpid heartbeating to 0.0.0.0/0.0.0.0:0" daemon prio=5 tid=259 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Object.wait(Native Method)
              at org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.waitTillNextIBR(IncrementalBlockReportManager.java:158)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:738)
              at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:878)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-47"  prio=10 tid=255 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-27"  prio=10 tid=235 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-11"  prio=10 tid=219 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-26"  prio=10 tid=234 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-39"  prio=10 tid=199 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-1"  prio=10 tid=209 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-34"  prio=10 tid=242 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-11"  prio=10 tid=171 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-7"  prio=10 tid=167 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "AsyncAppender-Dispatcher-Thread-51" daemon prio=5 tid=87 in Object.wait()
      java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at org.apache.log4j.AsyncAppender$Dispatcher.run(AsyncAppender.java:548)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-3-41"  prio=10 tid=249 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "ForkJoinPool-2-worker-29" daemon prio=5 tid=150 in Object.wait()
      java.lang.Thread.State: WAITING (on object monitor)
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
              at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
              at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
      "nioEventLoopGroup-2-29"  prio=10 tid=189 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "nioEventLoopGroup-2-26"  prio=10 tid=186 timed_waiting
      java.lang.Thread.State: TIMED_WAITING
              at java.lang.Thread.sleep(Native Method)
              at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.lang.Thread.run(Thread.java:748)
      "ForkJoinPool-2-worker-25" daemon prio=5 tid=146 in Object.wait()
      java.lang.Thread.State: WAITING (on object monitor)
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
              at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
              at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
      "surefire-forkedjvm-command-thread" daemon prio=5 tid=18 runnable
      java.lang.Thread.State: RUNNABLE
              at java.io.FileInputStream.readBytes(Native Method)
              at java.io.FileInputStream.read(FileInputStream.java:255)
              at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
              at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
              at java.io.DataInputStream.readInt(DataInputStream.java:387)
              at org.apache.maven.surefire.booter.MasterProcessCommand.decode(MasterProcessCommand.java:115)
              at org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:390)
              at java.lang.Thread.run(Thread.java:748)
      "ForkJoinPool-2-worker-4" daemon prio=5 tid=149 in Object.wait()
      java.lang.Thread.State: WAITING (on object monitor)
              at sun.misc.Unsafe.park(Native Method)
              at java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
              at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
              at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
      
      
      	at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:401)
      	at org.apache.hadoop.hdfs.server.datanode.TestBPOfferService$1.blockUtilSendFullBlockReport(TestBPOfferService.java:295)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.reRegister(BPServiceActor.java:927)
      	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:676)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processCommand(BPServiceActor.java:1351)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.lambda$enqueue$2(BPServiceActor.java:1397)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processQueue(BPServiceActor.java:1325)
      	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.run(BPServiceActor.java:1312)
      

      Attachments

        Issue Links

          Activity

            People

              ahussein Ahmed Hussein
              ahussein Ahmed Hussein
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 40m
                  1h 40m